Wednesday 10 November 2021

Python Panda Find and Replace a value

For the issue I was trying to solve, I knew which Column this value would appear in.  It is also the entire contents of the cell that I need to replace. 

After searching for 'python panda find and replace a value' I found this post  https://www.geeksforgeeks.org/python-pandas-dataframe-replace/ 

This suggests 

df.replace(to_replace ="Text I don't want",
                 value ="Text I do want")

However this didn't work for me.  What did work was to name the column specifically and to use 'str.replace', as mentioned here 

https://stackoverflow.com/questions/53141072/replacing-column-data-with-pandas-replace-not-working

uk['my_col'] = uk['my_col'].str.replace("Text I don't want", "Text I do want")

Wednesday 3 November 2021

Importing .Csv data into a table in Sequel Pro.

First you need to CREATE your table. 

Now click on the Table name on the left hand site of the Sequel Pro console. 

Next, select File > Import from the top Menu bar. 


From here on you'll have a wizard that'll talk you through the column matching. 


And that's it. 



Thursday 21 October 2021

DigitalOcean - FATAL: remaining connection slots are reserved for non-replication superuser connections.

 We were getting this error on an App running on Digital Ocean.  

This is what you'll find if you Google the issue 

https://www.digitalocean.com/community/questions/fatal-remaining-connection-slots-are-reserved-for-non-replication-superuser-connections

You’ll need to Edit the PostgreSQL configuration file and increase the value for maxconnections. The value for sharedbuffers may also need to be increased.

You may see here how more connections have been created:


This setting would be changed at →

  1. Login to DigitalOcean – The developer cloud

  2. Click on ‘Databases’

  3. Click on your DB

  4. Click on the ‘Connection Pools’ tab


This led us to page showing that we have 22 connections that we could use.  So why are we using all our connection slots on digital ocean. 

AND OUR SITUATION IT WAS THE CODE AT FAULT. 

In the code there were 13 connections to the DB being opened but they were not being closed. 
What I needed to do is add the db.dispose() after the connections been used. 


db_engine = create_engine('[CONNECTION_DETAILS]', pool_pre_ping=True)
...
db_engine.dispose()


Friday 15 October 2021

Macbook Pro M1 Chip and Docker issues and a Unresolved host.docker.internal issue.

Much of my frustration and wasted time at the moment can be attributed to setting up Docker. It’s probably worth noting that probably not all of it can be attributed to the Docker and M1 chip. But some of it certainly is.

Like with many of my issues I tend to blame myself and my own knowledge, but sometimes it’s not all my fault !! and if you have a M1 chip Mac it’d probably be worth looking into the potential pit falls first be commencing or purchasing one. As well as docker there has also been problems with Homebrew and Visual Studio Code. Of which VSC seems to be resolved now.


The first thing to note is that Docker have released an Apple Silicon Chip version, downloading this is the first thing that will save some time. Docker Desktop for Apple silicon And make sure you have the latest version.

Even with that there are also some on going ‘Known Issues’

  • Not all images are available for ARM64 architecture. You can add --platform linux/amd64 to run an Intel image under emulation. In particular, the mysql image is not available for ARM64. You can work around this issue by using a mariadb image.

    However, attempts to run Intel-based containers on Apple Silicon machines under emulation can crash as qemu sometimes fails to run the container. In addition, filesystem change notification APIs (inotify) do not work under qemu emulation. Even when the containers do run correctly under emulation, they will be slower and use more memory than the native equivalent.

    In summary, running Intel-based containers on Arm-based machines should be regarded as “best effort” only. We recommend running arm64 containers on Apple Silicon machines whenever possible, and encouraging container authors to produce arm64, or multi-arch, versions of their containers. We expect this issue to become less common over time, as more and more images are rebuilt supporting multiple architectures.

Pulling db (mysql:5.7)...
5.7: Pulling from library/mysql
ERROR: no matching manifest for linux/arm64/v8 in the manifest list entries

Meaning that in my Docker-compose file I had to add the following for the DB

1 db: 2 platform: linux/x86_64 3 image: mysql:5.7


At which point I can get to the database through Sequel Ace, however there’s no database ( this used to be maintained but now I seem to have to import when ever the site is fully torn down )

At this point I have my first site working at http://site1:8000/ and we can also use the Postman collection that comes with it. I say first site because for this project I need to get 2 sites working ( 2 independent github projects with their own docker-compose.yml files ). That second site will be using the endpoints that I just mentioned in the Postman collection.

Again I received an error on the ‘mysql’ image.

failed to solve with frontend dockerfile.v0: failed to create LLB definition: no match for platform in manifest sha256:5064###: not found
ERROR: Service 'mysql' failed to build : Build failed
davidmillward@Davids-MacBook-Air siaf %

So again we need to add this code to the DB image. In fact this time it’s ‘mysql’

1 mysql: 2 platform: linux/x86_64

And once again we’ll need to connect to the Mysql, and again it’s empty and we need to import the Database.

takes a bit of time this one :disappointed: . On this occasion so long it took me into the evening. Meaning that for the next step I would be restarting the Docker Daemon.

TIME SO FAR TAKEN 2h 30m


On the restart first thing the next day I’ve click play on all the images. However the site on http://site1:8000/ is no longer working, so I’ll need to investigate that.

The database is working though, so let’s shut down the other containers and see if that works.

That didn’t work, so lets ‘docker-compose down’ on all the containers and just concentrate on getting Site1 running again.

So I ‘docker-compose down’ , purged it in the debugging section of Docker UI and run the ‘docker-compose up -d’ . The database need doing again ( not too long this one. ) And now the first site works again , yay!

Only trouble is I’ll need to the DB on the second site again ( because of the purge ), but at least it’s first thing in the morning, so I won’t have to restart my laptop.
… see you in a bit ….!

TIME SO FAR TAKEN 4h 30m

So now we get to my issue which is that I can’t get the 2 sites to talk to each other internally.

This is the URL I used previously which worked fine.
http://host.docker.internal:8000/

If you want to find out why this method is needed and used then this blog explains this How to connect to the Docker host from inside a Docker container?



What do we know about this issue

It is mentioned on this page Docker Desktop for Apple silicon

Fixes since the Apple Silicon preview 7

  • The host.docker.internal and vm.docker.internal DNS entries now resolve.

The issue is mentioned here with no resolution Tunnel access is not working in docker container M1LIFECYCLE/STALE

Not supported according to this post https://betterprogramming.pub/macbook-m1-breaks-docker-development-14566ab6fa2e


Other things that you may ask me.

What comes back in the response ? It comes back empty but with no error.

Can you ping the FafeApi Container from the Drupal one ? Yes

Here’s the process to check that.

1exec into the drupal container and do a ping host.docker.internal

Although I needed to

1apt-get update 2apt-get install iputils-ping

This shows that I can ping the site ok.   So this issue remains unresolved. If anyone knows how to fix it then I'd really like to know. Much appreciated. :)

Thursday 23 September 2021

Mailgun: sending variables to template example - Fix ''Create and start coding'' greyed out issue.

First of all in Mailgun we'll add a template.   ( Keep reading for the ''Create and start coding'' greyed out issue fix tip ) 

So log into your account and on the dashboard you'll see all of your domains.  Click on the name of the domain that you want to add a template for.   Although there is a domain dropdown on the Template page itself, so no big deal if you want to change that later on. 

Next up, you'll need to click on 'Sending' in the left hand menu to see more options.  The 5th option down in this dropdown should be 'Templates' .



Click on this.  And then in the next page you'll get to 'Create Message template' 

You'll need to make sure the characters in the 'name' are valid and you have some sort of description.  Otherwise the 'Create and start coding' button is greyed out and you can't click on it. 


I’m sure this will trip someone else up , but the name input cannot be capitalised. If it is the Mailgun 'Create Message template' will remain greyed out.

In the HTML of template you’re setting up , you can add variable using the following.


1{{ firstname }} {{ lastname }}


How to send from Flask .

So here the function for sending a standard MIME email. With some HTML

1 def send_html_message(self): 2 return requests.post( 3 self.send_message_api, 4 auth=("api", self.mailgun_api_key), 5 data={ 6 "from": self.from_string, 7 "to": [self.to_address_2], 8 "cc": [self.to_address_1], 9 "subject": "HTML test", 10 "html": "<strong>Hello World</strong><br><p>Hi</p><p>This is an HTML test send </p>", 11 }, 12 )

 

And this is one where we use our ‘newsletter’ template and send through some variables.


1 def send_simple_message_template(self): 2 return requests.post( 3 self.send_message_api, 4 auth=("api", self.mailgun_api_key), 5 data={"from": self.from_string, 6 "to": "David Millward <dj@flowmo.co>", 7 "subject": self.subject, 8 "template": "newsletter", 9 "v:firstname":"David", 10 "v:lastname":"John", 11 })


This one will send our newsletter, with the clients name on it.