The challenge contains 4 required tasks

1. Run a simple Dataflow job
2. Run a simple Dataproc job
3. Run a simple Dataprep job
4. Complete one of the AI tasks

1.Run a simple Dataflow job

bq mk lab

gsutil cp gs://cloud-training/gsp323/lab.csv .

cat lab.csv

gsutil cp gs://cloud-training/gsp323/lab.schema .

cat lab.schema

2. Run a simple Dataproc job

• You have used Dataproc in the quest, now you must run another example Spark job using Dataproc.

• Before you run the job, log into one of the cluster nodes and copy the /data.txt file into hdfs (use the command hdfs dfs -cp gs://cloud-training/gsp323/data.txt /data.txt)

• Run a Dataproc job using the values below

Wait for the job to finish then click on check progress.

3. Run a simple Dataprep job

• You have used Dataprep to import data files and transformed them to gain views of the data. Use Dataprep to import one CSV file (described below) that holds data of lab executions.

gs://cloud-training/gsp323/runs.csv structure:

Perform the following transforms to ensure the data is in the right state:

• Remove all rows with the state of "FAILURE"
• Remove all rows with 0 or 0.0 as a score (Use the regex pattern /(^0$|^0.0$)/)
• Label columns with the names above

Note: Your Dataprep job should look similar to the following

Make sure you run the job. You will need to wait until the Dataflow job completes before you can grade this task.

4. AI
Complete one of the following tasks below.

• Use Google Cloud Speech API to analyze the audio file gs://cloud-training/gsp323/task4.flac. Once you have analyzed the file you can upload the resulting analysis to Cloud Speech Location.

• Use the Cloud Natural Language API to analyze the sentence from text about Odin. The text you need to analyze is "Old Norse texts portray Odin as one-eyed and long-bearded, frequently wielding a spear named Gungnir and wearing a cloak and a broad hat." Once you have analyzed the text you can upload the resulting analysis to Cloud Natural Language Location.

• Use Google Video Intelligence and detect all text on the video gs://spls/gsp154/video/train.mp4. Once you have completed the processing of the video, pipe the output into a file and upload to Video Intelligence Location . Ensure the progress of the operation is complete and the service account you're uploading the output with has the Storage Object Admin role

gcloud iam service-accounts create my-natlang-sa \
  --display-name "my natural language service account"

gcloud iam service-accounts keys create ~/key.json \
  --iam-account my-natlang-sa@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com

export GOOGLE_APPLICATION_CREDENTIALS="/home/$USER/key.json"

gcloud auth activate-service-account my-natlang-sa@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com --key-file=$GOOGLE_APPLICATION_CREDENTIALS

gcloud ml language analyze-entities --content="Old Norse texts portray Odin as one-eyed and long-bearded, frequently wielding a spear named Gungnir and wearing a cloak and a broad hat." > result.json

gcloud auth login
(Copy the token from the link provided)


gsutil cp result.json gs://YOUR_PROJECT-marking/task4-cnl.result

Create an API key and export as API_KEY variable

export API_KEY={Replace with API KEY}

nano request.json

Add this content:

{
  "config": {
      "encoding":"FLAC",
      "languageCode": "en-US"
  },
  "audio": {
      "uri":"gs://cloud-training/gsp323/task4.flac"
  }
}

TERMINAL

curl -s -X POST -H "Content-Type: application/json" --data-binary @request.json \
"https://speech.googleapis.com/v1/speech:recognize?key=${API_KEY}" > result.json

gsutil cp result.json gs://YOUR_PROJECT-marking/task4-gcs.result


gcloud iam service-accounts create quickstart

gcloud iam service-accounts keys create key.json --iam-account quickstart@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com

gcloud auth activate-service-account --key-file key.json

export ACCESS_TOKEN=$(gcloud auth print-access-token)


nano request.json