Django - Grabbing a JSON object from QueryDict

 
What I'm doing here is passing some JSON through the 'rest.framework' .  The problem is that when using 'request.json'  I'm seeing the following being returned. 

<QueryDict: {'': ['{"time":{"0":"06:00","1":"07:00","2":"08:00","3":"09:00","4":"10...

What I need to do is extract the JSON out .  To do this , the following works. 

list(data.values())[0]

M1 Chip issue with AWS Lambda SAM Layers

 

Just a quick note , if you've searched and found this blog then I expect you already know deep down that the M1 chip is your issue.  There is a solution though.  Let me explain. 

So, what we've been doing is using Lambda Layers to install Python packages on our Lambda installation.  However, when putting the same code on my Mac ( M1 Chip ) it doesn't work ! 

https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html

It does work if we take the zip file from the AMD64 chipped Mac , and put that on my Mac and proceed. 

That is the solution we're having to use at the moment , the zip needs to be compiled on the AMD64 chip , doing it on the M1 chip won't work. 

Could not 'pipenv install email' - 'python setup.py egg_info did not run successfully.' error.

Here's the error code I was getting. 

Installing email... Error: An error occurred while installing email! Error text: Collecting email Using cached email-4.0.2.tar.gz (1.2 MB) Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'error' error: subprocess-exited-with-error × python setup.py egg_info did not run successfully. │ exit code: 1 ╰─> [1 lines of output] ERROR: Can not execute `setup.py` since setuptools is not available in the build environment. [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed × Encountered error while generating package metadata. ╰─> See above for output. note: This is an issue with the package mentioned above, not pip. hint: See above for details. This is likely caused by a bug in email. Report this to its maintainers. ✘ Installation Failed

The recommended solution across the web seemed to be to run the following. 

pipenv install --upgrade setuptools

However, this didn't work for me.  I needed to use the '--pre' tag. 

pipenv install --pre email


This now works, the version I had the issue with was 'email-4.0.2'


So, if you have a 'email-4.0.2 won't pipenv install' issue then I hope this post helps. 

Wordpress installation on Docker - How to access the Database

 After running through this blog on how to create a Wordpress site on Docker.  ( Which works excellently ) https://davidyeiser.com/tutorials/docker-wordpress-theme-setup

After doing so, I couldn't quite work out what details I needed to use in Sequel Pro / Sequel Ace. 

There's a line that I needed to add to the Wordpress DB setup though, and that was to add the ports .

ports:

- "3366:3306"

So my entire file looks like this. 

version: '3'

services:

db:

image: mysql:5.7

volumes:

- db_data:/var/lib/mysql

ports:

- "3366:3306"

restart: always

environment:

MYSQL_ROOT_PASSWORD: somewordpress

MYSQL_DATABASE: wordpress

MYSQL_USER: wordpress

MYSQL_PASSWORD: wordpress

wordpress:

depends_on:

- db

image: wordpress:5.1.1-php7.3-apache

ports:

- "8000:80"

restart: always

environment:

WORDPRESS_DB_HOST: db:3306

WORDPRESS_DB_USER: wordpress

WORDPRESS_DB_PASSWORD: wordpress

working_dir: /var/www/html

volumes:

- ./wp-content:/var/www/html/wp-content

- ./uploads.ini:/usr/local/etc/php/conf.d/uploads.ini

volumes:

db_data:

And then I can use

Vue.js not building correctly in 'dist' folder 'npm run build'

 'npm run build' 'dist' folder not working.

I'd been hitting a bit of a brick wall with this task, so thought it'd be a good idea to commit it to the blog. Incase someone else finds it an can save themselves a heap of time. The task I was trying to achieve was

'How to host a Vue.js project on Firebase'  .  

And I was following this blog and youtube video for guidance. 

How to deploy vue.js applications with firebase hosting

And Net Ninja's 

Firebase Hosting Tutorial #4 - Deploying a Vue Site

But on a site ( that was already built and working locally ) I could only see the Firebase hosting holding message . 

'Firebase Hosting Setup Complete' 

I could however could create a new Vue.js website ( boiler plate ) and work from that, but building up from scratch again would be excruciating. 

Let's take a look at process I was following and where it was getting stuck. 

$. firebase init

Scroll down to the first 'Hosting config' choice and click on 'spacebar' and then 'return'. 


There is already a public folder in use, so we'll want to set the public directory as 'dist' 

As we're using Vue.js which is a javascript app that rewrites all the pages through the index page, then we need to select 'y' in response to , rewrite all urls to /index.html . This creates a dummy index page.

We don’t want to deploy this , we now want to build our vue application so that all the files go inside this ‘dist’ folder and then we want to deploy it.


1 npm run build

And we should now be able to view our site with.

1firebase serve

 

And this gives us the same ‘Firebase Hosting’ message we’re seeing before

On my basic vue.js install though, I could see that it was working fine. One thing that caught my eye was that the 'dist' folder was still empty. Hold on ! No, the issue was another 'dist' folder had been created and I had one in the 'root' of the project and one in the 'app' folder.

1. I removed both the ‘dist’ folders and started again.

2. Ran ‘fiebase init’ and chose the hosting as before.

3. The ‘dist’ folder was created in Root . But I manually moved it to the ‘app’ folder

4. I then ran ‘npm run build’ . this created the new files.

5. In ‘firebase.json’ I had to add the following code
   
   

   1 "hosting": { 2 "public": "app/dist",

    

6. Ran ‘firebase serve’.

   
   So I have the site showing using ‘firebase serve’ .

1. 
firebase deploy --only hosting

And I can now see it on the app online :) I hope this saves someone a heap of time, as it took me a while to work this one out :/

CodeRed CMS - How to Order ( weight ) the Navigation snippets

 The task was simple, I'd added a load of menu items to the top menu using snippets but I wanted to change the order 

What no drag and drop !  And neither could I see an input box to enter a position.   Surely I wouldn't have to get my hands dirty and write some code for this. 

Here's an easy fix though.  

As the navigation uses flexbox .  All you need to do is call the Bootstrap Order class.

And then in your 'navigation' item 'snippet' just add the ordering like this

Gsutil - How to empty a GCS bucket

 Although I  couldn't find a 'gsutil' command to empty a bucket as such the following will remove and then we add the bucket. 

I've also add the code here for how to do this from an Airflow DAG . 

% gsutil -m rm -r gs://djem_photos

% gsutil -m mkdir gs://djem_photos

from airflow.operators import bash_operator bucket = 'gs://temp_gp_tasks'

with models.DAG(

'remove_files_from_gcs',

schedule_interval=None,

default_args=default_dag_args,

catchup=True) as dag:

remove_gcs_bucket = bash_operator.BashOperator(

task_id='remove_gcs_bucket',

bash_command='gsutil -m rm -r {bucket}'.format(

bucket=bucket))

recreate_gcs_bucket = bash_operator.BashOperator(

task_id='recreate_gcs_bucket',

bash_command='gsutil -m mkdir {bucket}'.format(

bucket=bucket))

remove_gcs_bucket >> recreate_gcs_bucket

Airflow Connection Code Example - Access connection data from code

 Here's a straight forward piece of code, but I found this solution difficult to find and uncover .  So here's the solution to getting the data from the Airflow connection in to my DAG - in Google Cloud Platform 

from airflow.hooks.base_hook import BaseHook

conn = BaseHook.get_connection('my_conn')

def connect_to_somewhere():

logging.info(conn.login)

logging.info(conn.password)

Hope that helps someone out there ! 

GCP DAGS " Given * file, /home/airflow/dags/* , could not be opened.

Although this post is about solving my specific issue with locating the 'googleads.yaml' and it's 'private_key' file , it's will also be useful to others who want to Access any other file that they have uploaded. 

I had the following code that I could not get to work.  


FOLDER_PATH = '~/dags/' 2 3def gam_traffic(): 4 client = ad_manager.AdManagerClient.LoadFromStorage(FOLDER_PATH + 'googleads.yaml')

  I'd tried other FOLDER names , like the gs:// bucket address and the full address like

europe-west1-composer-XXX-XXXX-bucket.  All to no avail. 

RED HERRING ALERT

Using the gs:// bucket address worked fine for Saving and Retrieving .csv files using Python Panda, but I couldn’t get the address to work when using the Python Open command. My assumption now is that the Python Panda library must contain some ‘magic’ that process this address when it see that the URI starts with ‘gs://’

To solve my issue and find the path I needed then I ran the following tasks to prove what folder we are in , and all the files and folders in it.


import os 2 3# Print working dir 4def print_working_dir(): 5 directory = os.getcwd() 6 # Iterating through the json 7 logging.info(directory) 8 return "success"

 

1import glob 2 3# Print working dir 4def list_working_dir(): 5 for filename in glob.iglob("./**/*", recursive=True): 6 logging.info(filename) 7

The second task takes ages to run, as its iterating through loads of folders. In our case we intercepted the logs while it was running , as we could already see the information that we needed.

In our case the filepath we needed was

1./gcsfuse/dags/googleads.yaml'