Trying to migrate SQLite DB to PostgresDB docker container.

[kokomo123]

Description

I expected it to import / migrate fine but it's saying its missing the shadow banned part of the database. I don't have anyone shadow banned so i wish i could just make it ignore what its missing. I'm honestly not too sure if its my db, me, synapse or what. Did i corrupt something?

Steps to reproduce

1.Pull the docker-compose structure up
2.Enter in the details in homeserver-postgres.yaml and enter into the synapse docker container.
3. go into data and run the migrate command with
homeserver.db and homeserver-postgres.yaml using

synapse_port_db --sqlite-database homeserver.db.snapshot \
    --postgres-config homeserver-postgres.yaml

Version information

• Homeserver: matrix.kokomo.cloud

• Version: 1.19.1

• Install method: docker using docker-compose

• Platform: Debian 10 x64 in a vm, synapse is running in a container and postgres db is in a container along with element.

Here's my log from port-synapse:
port-synapse.log

[clokep]

The shadow-banning feature will be added in 1.20.0, can you paste the actual error you're seeing?

[kokomo123]

In port-synapse if you scroll all the way to the bottom, that should be the error but heres a screenshot if this helps. It hangs on checking for port progress.

As well as a pastebin: https://pastebin.com/yGbHSBWe

[clokep]

I had missed you put up logs, sorry about that!

How are you running the script? Is it from a checkout from git or...?

[kokomo123]

It's from the docker container itself somehow probably built in. I did checkout from git the whole repo though. so probably?

[clokep]

It's from the docker container itself somehow probably built in. I did checkout from git the whole repo though. so probably?

What branch are you using?

It seems like the sqlite database is running from a version of the database from Synapse 1.19.1, but the script is running expecting a version of 1.20.0 (which isn't yet released).

I'd make sure that the release-v1.19.1 tag is checked out and build the docker image again.

[kokomo123]

Well i did pull from a earlier version before 1.19.1.. Can i keep my homeserver.yaml and homeserver.db and then pull the repo again? Docker has the latest tag, so i guess that's why the image / script is expecting 1.20.0
Also im not so sure which branch i pulled from... rip.

[kokomo123]

So perhaps this is solved. I'll clone from v1.19.1 and remove the old folder, keeping in mind to preserve homeserver.yaml and homeserver.db and see where it gets us. If it doesn't work, ill be sure to make a new issue / reopen this one 👍🏻

[clokep]

Sounds good! If it doesn't work please just report back here! 👍

[kokomo123]

Hey im back again. Seems like the issue wasn't resolved. When i tried forking using git clone --branch v1.19.1 [the git repo] it just seems to not generate its homeserver.yaml, thus preventing me from doing anything. I tried deleting the synapse folder, even tried a LXC (Linux container) this morning with the package method. Still refuses my homeserver.db because it just crashes with sqlite. I may be doing something wrong honestly, but I tried the best that i could. The nuclear option could just be my final resort at this point just wipe everything and recreate the db from scratch and try to hook it up via postgres, but im afraid it'll just refuse to connect still.

[richvdh]

It sounds like something is very confused here. You're talking about checking things out from git, but also about using docker. You need to pick a method and stick with it, otherwise, as you've discovered, you're going to have problems where you have mismatched versions.

When i tried forking using git clone --branch v1.19.1 [the git repo] it just seems to not generate its homeserver.yaml

Cloning a git repo does not automatically generate a homeserver.yaml. But that shouldn't matter, because if you are trying to migrate to postgres, you must already have a homeserver.yaml.

If you have a working docker container, then use the synapse_port_db script from that. Sadly, the docker image was built by a community contributor and let's just say that documentation wasn't their strong point. I've opened #8304 about this.

If you still have problems, you can report back here, but please be clear what exactly you are doing, and copy any errors you see.

[kokomo123]

@richvdh Sorry im a hard explainer. I was trying it via 2 different methods, should of sticked to one for sure. synapse_port_db doesn't work because the script errors out since it's expecting 1.20.0, but im on 1.19.1. The git repo was in order to get docker working, the generate command kept on failing because i wanted it to point to a specific location, not in /data. Im not sure how i got it mounted in /files before. That's what i was trying to do. And yes, probably the documentation will help me in the future.

[kokomo123]

Closing issue again because it's probably my fault for cloning this github repo on develop, not knowing that i should clone a specific branch instead. I'll have to redo everything from scratch. Thank you for trying to help :)