Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Very slow replication for databases with attachments #1125

Closed
sergey-safarov opened this issue Jan 24, 2018 · 11 comments
Closed

Very slow replication for databases with attachments #1125

sergey-safarov opened this issue Jan 24, 2018 · 11 comments
Labels
bug waiting on user

Comments

@sergey-safarov
Copy link

Expected Behavior

Between bigcouch host replication take not more 3-4 hours for 1500 databases (25Gb total size).
I expects replication from couchdb 2.0 host to bigcouch will be similar

Current Behavior

Some databases cannot be replicated for several hours.

Possible Solution

Not knows

Steps to Reproduce (for bugs)

  1. Create 1500 databases about 25Gb total size on couchdb 2.0 hosts;
  2. Start replication to bigcouch host from couchdb, curl command is send to couchdb host;
  3. Check Active Tasks screen on coucudb host.

Context

I want replicate database to other host.

Your Environment

  • Version used: 2.1.1
  • Browser Name and version: Google Chrome Version 63.0.3239.132 (Official Build) (64-bit)
  • Operating System and version (desktop or mobile): CentOS 7 on bare metal server.
  • Link to your project: none
@sergey-safarov
Copy link
Author

Example of _active_tasks output (formated)

[  
   {  
      "node":"couchdb@127.0.0.1",
      "pid":"<0.23837.50>",
      "changes_pending":76072,
      "checkpoint_interval":30000,
      "checkpointed_source_seq":"1-g1AAAAF1eJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfaE1TmA1MWD1GUwJzLmAnnsiWmmhhZpydj04DMpAWRSPUEb81iAJEMDkAIqnU-M2gUQtfuJUXsAovY-MWofQNSC3JsFAGwAZt4",
      "continuous":false,
      "database":null,
      "doc_id":null,
      "doc_write_failures":0,
      "docs_read":27,
      "docs_written":27,
      "missing_revisions_found":27,
      "replication_id":"83de18e815b496003e016afaca85d7bc+create_target",
      "revisions_checked":27,
      "source":"http://217.12.247.99:5984/account%2f63%2ff8%2f14795fd5b8c74aa04ee865c52f73-201712/",
      "source_seq":"527-g1AAAAJjeJyd0L0KwjAQAOBgBV19AFFfQJK2adrJOjmIk7s2f6WUqpOzvom-ib6Jvkm9JB2L0BK4wB33cXcVQmhceApN5OUqCyVS4rMlhkcqKA04ErO6rsvC4-sTJEYS6yCmrK3hDyPmEMWqkQ5W8jVnWJGuUmqkYyMtrMRzSuJcdpUyI90aaWolnBCpVdftzkOI6A4fYA-jbawmIhUKzXtpT6e9jLa3WsIopoHqpb2d9jHazt1MRpFUfi_t6zR7t62bLQ5ZjltnK3-ySqGL",
      "started_on":1516774642,
      "target":"http://217.12.247.98:5984/account%2f63%2ff8%2f14795fd5b8c74aa04ee865c52f73-201712/",
      "through_seq":"1-g1AAAAF1eJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfaE1TmA1MWD1GUwJzLmAnnsiWmmhhZpydj04DMpAWRSPUEb81iAJEMDkAIqnU-M2gUQtfuJUXsAovY-MWofQNSC3JsFAGwAZt4",
      "type":"replication",
      "updated_on":1516776894,
      "user":null
   },
   {  
      "node":"couchdb@127.0.0.1",
      "pid":"<0.2536.290>",
      "changes_pending":null,
      "checkpoint_interval":30000,
      "checkpointed_source_seq":0,
      "continuous":false,
      "database":null,
      "doc_id":null,
      "doc_write_failures":0,
      "docs_read":4,
      "docs_written":4,
      "missing_revisions_found":4,
      "replication_id":"880f88e653a7900c1e7c43461eff471d+create_target",
      "revisions_checked":4,
      "source":"http://217.12.247.99:5984/account%2fa4%2f87%2f97afcbf4c1782d55613a5f86b0bf-201712/",
      "source_seq":"4-g1AAAAHbeJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfYgdRnMiYy5QB67RXJKinGKITY9-ExyAJkUj2KSobFpslGqGakmJYBMqifo9jwWIMnQAKSASucTo3YBRO1-hAvTzJMtTY2SSHQhxLQDENPuI0xLTTZLM0gjNeQgpj2AmAbycxYAxMp_Nw",
      "started_on":1516776088,
      "target":"http://217.12.247.98:5984/account%2fa4%2f87%2f97afcbf4c1782d55613a5f86b0bf-201712/",
      "through_seq":"4-g1AAAAHbeJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfYgdRnMiYy5QB67RXJKinGKITY9-ExyAJkUj2KSobFpslGqGakmJYBMqifo9jwWIMnQAKSASucTo3YBRO1-hAvTzJMtTY2SSHQhxLQDENPuI0xLTTZLM0gjNeQgpj2AmAbycxYAxMp_Nw",
      "type":"replication",
      "updated_on":1516776089,
      "user":null
   },
   {  
      "node":"couchdb@127.0.0.1",
      "pid":"<0.29148.294>",
      "changes_pending":null,
      "checkpoint_interval":30000,
      "checkpointed_source_seq":0,
      "continuous":false,
      "database":null,
      "doc_id":null,
      "doc_write_failures":0,
      "docs_read":4,
      "docs_written":4,
      "missing_revisions_found":4,
      "replication_id":"1aebc296b560ed25c8153cda8f55069e+create_target",
      "revisions_checked":4,
      "source":"http://217.12.247.99:5984/account%2fa4%2f87%2f97afcbf4c1782d55613a5f86b0bf-201710/",
      "source_seq":"4-g1AAAAG5eJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfYgdRnMiUy5QB67pYGhYWKiGTY9-ExyAJkUT9jGBJC6eqiNjGAbzSwtUw1SDEm0MY8FSDI0ACmgYfPx2wpRuwCidj_CZsMkI0NLM1OybD4AMe0-MTY_gKgFhU0WAEpFdiQ",
      "started_on":1516776027,
      "target":"http://217.12.247.98:5984/account%2fa4%2f87%2f97afcbf4c1782d55613a5f86b0bf-201710/",
      "through_seq":"1-g1AAAAF1eJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfYgdRnMiYy5QB67pYGhYWKiGTY9-ExyAJkUT9jGBJC6eoLq8liAJEMDkAIqnU-M2gUQtfuJUXsAovY-MWofQNSC3JsFAMMyZnM",
      "type":"replication",
      "updated_on":1516776028,
      "user":null
   },
   {  
      "node":"couchdb@127.0.0.1",
      "pid":"<0.31083.294>",
      "changes_pending":null,
      "checkpoint_interval":30000,
      "checkpointed_source_seq":0,
      "continuous":false,
      "database":null,
      "doc_id":null,
      "doc_write_failures":0,
      "docs_read":4,
      "docs_written":4,
      "missing_revisions_found":4,
      "replication_id":"c125005084bccdfaffc089a2d25cdd43+create_target",
      "revisions_checked":4,
      "source":"http://217.12.247.99:5984/account%2fa4%2f87%2f97afcbf4c1782d55613a5f86b0bf-201711/",
      "source_seq":"4-g1AAAAGXeJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____szKYE5lygQLsSSmWJoYWKdg04DEmSQFIJtmjmGRoamCSaG5AqkkOIJPiQSYlMuBTlwBSV09QXR4LkGRoAFJApfOJUbsAonY_MWoPQNTeJ0btA4hakHuzABv8bmY",
      "started_on":1516776055,
      "target":"http://217.12.247.98:5984/account%2fa4%2f87%2f97afcbf4c1782d55613a5f86b0bf-201711/",
      "through_seq":"1-g1AAAAF1eJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfYgdRnMiYy5QB67oamBSaK5ATY9-ExyAJkUT9jGBJC6eoLq8liAJEMDkAIqnU-M2gUQtfuJUXsAovY-MWofQNSC3JsFAI3AZkI",
      "type":"replication",
      "updated_on":1516776057,
      "user":null
   },
   {  
      "node":"couchdb@127.0.0.1",
      "pid":"<0.32724.392>",
      "changes_pending":null,
      "checkpoint_interval":30000,
      "checkpointed_source_seq":0,
      "continuous":false,
      "database":null,
      "doc_id":null,
      "doc_write_failures":0,
      "docs_read":0,
      "docs_written":0,
      "missing_revisions_found":0,
      "replication_id":"7a796b343b45bb41d6cb9a76e6f9e675+create_target",
      "revisions_checked":0,
      "source":"http://217.12.247.99:5984/account%2fc3%2f43%2f646ab95638f945c8ec895263cf2e-201707/",
      "source_seq":0,
      "started_on":1516776898,
      "target":"http://217.12.247.98:5984/account%2fc3%2f43%2f646ab95638f945c8ec895263cf2e-201707/",
      "type":"replication",
      "updated_on":1516776898,
      "user":null
   },
   {  
      "node":"couchdb@127.0.0.1",
      "pid":"<0.7273.48>",
      "changes_pending":null,
      "checkpoint_interval":30000,
      "checkpointed_source_seq":0,
      "continuous":false,
      "database":null,
      "doc_id":null,
      "doc_write_failures":0,
      "docs_read":8,
      "docs_written":8,
      "missing_revisions_found":8,
      "replication_id":"260c79c446f46814cdf361b5d02b59ce+create_target",
      "revisions_checked":8,
      "source":"http://217.12.247.99:5984/account%2f63%2ff8%2f14795fd5b8c74aa04ee865c52f73-201711/",
      "source_seq":"1041-g1AAAAJjeJyd0LsNwjAQBmCLIEHLBDABivOyXZEdGABsn60oBKhoKICahhFgE1gEwSbBj5QRUqKT_pPudF9xFUJoXASAJnJ_kAWIHEdkHprClVkNOBLTuq7LIuBqawYjpkKuCGs7-MOImUmxaKSTkwSVOo1oVym30qqRjk4CAlpy6CqtrXRupKuXFERSdJV2Q5PoYprB7la7OY2zBHSGe2kPrz2t9vb_yqiMdT_t5bWP1ZZOIxR4Eqte2tdr7m8bp2nMaJSKtrvyB9O6pI8",
      "started_on":1516774615,
      "target":"http://217.12.247.98:5984/account%2f63%2ff8%2f14795fd5b8c74aa04ee865c52f73-201711/",
      "through_seq":"3-g1AAAAGXeJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfaE1TmA1MUTVpcAUlcPUpfBnMiYC-Sxp6SmGCUnpWDTg8ekPBYgydAApICGzcdvK0TtAoja_cSoPQBRex_kSiawK80tUhJNjFPJcuUDiGmgsMkCAN8Hb1s",
      "type":"replication",
      "updated_on":1516774616,
      "user":null
   }
]

Replication to database account%2f63%2ff8%2f14795fd5b8c74aa04ee865c52f73-201711 is now acrive about 1 hour, but processes only 8 doc.

@sergey-safarov
Copy link
Author

After two hours

[  
   {  
      "node":"couchdb@127.0.0.1",
      "pid":"<0.23837.50>",
      "changes_pending":76072,
      "checkpoint_interval":30000,
      "checkpointed_source_seq":"1-g1AAAAF1eJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfaE1TmA1MWD1GUwJzLmAnnsiWmmhhZpydj04DMpAWRSPUEb81iAJEMDkAIqnU-M2gUQtfuJUXsAovY-MWofQNSC3JsFAGwAZt4",
      "continuous":false,
      "database":null,
      "doc_id":null,
      "doc_write_failures":0,
      "docs_read":878,
      "docs_written":878,
      "missing_revisions_found":878,
      "replication_id":"83de18e815b496003e016afaca85d7bc+create_target",
      "revisions_checked":878,
      "source":"http://217.12.247.99:5984/account%2f63%2ff8%2f14795fd5b8c74aa04ee865c52f73-201712/",
      "source_seq":"878-g1AAAAJjeJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____szKYE3NzgQLsyQapxham5tg04DEmSQFIJtlDTeoEm2SUmmhukGJIqkkOIJPioSb5g01KTDM1tEhLJtWkBJBJ9VCTjMEmGVgaJqemkOq7PBYgydAApICGzQeZVgk2LcksxSQpNZEs0xZATNsPMq0VbJqluamBqXEKWaYdgJh2H2RaPSTMks3MklOMyDLtAcS0_wifWlqYmKcZYHVbFgBFpqLq",
      "started_on":1516774642,
      "target":"http://217.12.247.98:5984/account%2f63%2ff8%2f14795fd5b8c74aa04ee865c52f73-201712/",
      "through_seq":"504-g1AAAAJjeJyd0EsKwjAQANBgBd16AFEvIEnbNC0IFlyqa5c2yaSUUnXlWm-iN9Gb6E1qPl0WoSUwgRnmMTMVQmhceIAm8nKVBYiU-GyJ9SOVLg04ErO6rsvC46uTTowkVkFMWVvDH0bMdRTrRjpYyVecYSBdpdRIx0ZaWInnlMS57CplRro10tRKOCFSQdftzkMd0V1_GnsYbWM1EUEoFO-lPZ32MtrOagmjmAbQS3s77WO0vbuZjCIJfi_t6zR7t62bLQ5ZjltnK3-QTqF0",
      "type":"replication",
      "updated_on":1516781845,
      "user":null
   },
   {  
      "node":"couchdb@127.0.0.1",
      "pid":"<0.29148.294>",
      "changes_pending":null,
      "checkpoint_interval":30000,
      "checkpointed_source_seq":0,
      "continuous":false,
      "database":null,
      "doc_id":null,
      "doc_write_failures":0,
      "docs_read":4,
      "docs_written":4,
      "missing_revisions_found":4,
      "replication_id":"1aebc296b560ed25c8153cda8f55069e+create_target",
      "revisions_checked":4,
      "source":"http://217.12.247.99:5984/account%2fa4%2f87%2f97afcbf4c1782d55613a5f86b0bf-201710/",
      "source_seq":"4-g1AAAAG5eJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfYgdRnMiUy5QB67pYGhYWKiGTY9-ExyAJkUT9jGBJC6eqiNjGAbzSwtUw1SDEm0MY8FSDI0ACmgYfPx2wpRuwCidj_CZsMkI0NLM1OybD4AMe0-MTY_gKgFhU0WAEpFdiQ",
      "started_on":1516776027,
      "target":"http://217.12.247.98:5984/account%2fa4%2f87%2f97afcbf4c1782d55613a5f86b0bf-201710/",
      "through_seq":"1-g1AAAAF1eJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfYgdRnMiYy5QB67pYGhYWKiGTY9-ExyAJkUT9jGBJC6eoLq8liAJEMDkAIqnU-M2gUQtfuJUXsAovY-MWofQNSC3JsFAMMyZnM",
      "type":"replication",
      "updated_on":1516776028,
      "user":null
   },
   {  
      "node":"couchdb@127.0.0.1",
      "pid":"<0.31083.294>",
      "changes_pending":5214,
      "checkpoint_interval":30000,
      "checkpointed_source_seq":"1-g1AAAAF1eJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfYgdRnMiYy5QB67oamBSaK5ATY9-ExyAJkUT9jGBJC6eoLq8liAJEMDkAIqnU-M2gUQtfuJUXsAovY-MWofQNSC3JsFAI3AZkI",
      "continuous":false,
      "database":null,
      "doc_id":null,
      "doc_write_failures":0,
      "docs_read":4,
      "docs_written":4,
      "missing_revisions_found":4,
      "replication_id":"c125005084bccdfaffc089a2d25cdd43+create_target",
      "revisions_checked":4,
      "source":"http://217.12.247.99:5984/account%2fa4%2f87%2f97afcbf4c1782d55613a5f86b0bf-201711/",
      "source_seq":"4-g1AAAAGXeJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____szKYE5lygQLsSSmWJoYWKdg04DEmSQFIJtmjmGRoamCSaG5AqkkOIJPiQSYlMuBTlwBSV09QXR4LkGRoAFJApfOJUbsAonY_MWoPQNTeJ0btA4hakHuzABv8bmY",
      "started_on":1516776055,
      "target":"http://217.12.247.98:5984/account%2fa4%2f87%2f97afcbf4c1782d55613a5f86b0bf-201711/",
      "through_seq":"1-g1AAAAF1eJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfYgdRnMiYy5QB67oamBSaK5ATY9-ExyAJkUT9jGBJC6eoLq8liAJEMDkAIqnU-M2gUQtfuJUXsAovY-MWofQNSC3JsFAI3AZkI",
      "type":"replication",
      "updated_on":1516779838,
      "user":null
   },
   {  
      "node":"couchdb@127.0.0.1",
      "pid":"<0.7273.48>",
      "changes_pending":235248,
      "checkpoint_interval":30000,
      "checkpointed_source_seq":"3-g1AAAAGXeJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfaE1TmA1MUTVpcAUlcPUpfBnMiYC-Sxp6SmGCUnpWDTg8ekPBYgydAApICGzcdvK0TtAoja_cSoPQBRex_kSiawK80tUhJNjFPJcuUDiGmgsMkCAN8Hb1s",
      "continuous":false,
      "database":null,
      "doc_id":null,
      "doc_write_failures":0,
      "docs_read":773,
      "docs_written":773,
      "missing_revisions_found":773,
      "replication_id":"260c79c446f46814cdf361b5d02b59ce+create_target",
      "revisions_checked":773,
      "source":"http://217.12.247.99:5984/account%2f63%2ff8%2f14795fd5b8c74aa04ee865c52f73-201711/",
      "source_seq":"1041-g1AAAAJjeJyd0LsNwjAQBmCLIEHLBDABivOyXZEdGABsn60oBKhoKICahhFgE1gEwSbBj5QRUqKT_pPudF9xFUJoXASAJnJ_kAWIHEdkHprClVkNOBLTuq7LIuBqawYjpkKuCGs7-MOImUmxaKSTkwSVOo1oVym30qqRjk4CAlpy6CqtrXRupKuXFERSdJV2Q5PoYprB7la7OY2zBHSGe2kPrz2t9vb_yqiMdT_t5bWP1ZZOIxR4Eqte2tdr7m8bp2nMaJSKtrvyB9O6pI8",
      "started_on":1516774615,
      "target":"http://217.12.247.98:5984/account%2f63%2ff8%2f14795fd5b8c74aa04ee865c52f73-201711/",
      "through_seq":"769-g1AAAAJjeJyd0EsKwjAQBuBgBd16Aj2BNOkj6crewYXgRjN5EEvVlWu9id5Eb6I3qXl04aIILQP_wAzzLaZGCE1NJNFMnC_CSCgxocvYFq7tasQRzJumqUzEt0c7mBQq5ooWXQd_GFjYhFUrKS8BEzojrK9UOmnXSsZLkkotuOwr7Z10baVDkJQkAvpKp7FNdLPNYnengdd4kUqd40HaI2jPn3_lTCR6mPYK2ttpa69RJnmaqEHaJ2j-bxuvaVwwkkHXXfUFtg-jfw",
      "type":"replication",
      "updated_on":1516781818,
      "user":null
   }
]

Same database. 773 docs written and 235248 pending changes.

@sergey-safarov
Copy link
Author

Same database after 6 hours, 2542 docs written and 234248 pending changes.

[  
   {  
      "node":"couchdb@127.0.0.1",
      "pid":"<0.29148.294>",
      "changes_pending":6843,
      "checkpoint_interval":30000,
      "checkpointed_source_seq":"1-g1AAAAF1eJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfYgdRnMiYy5QB67pYGhYWKiGTY9-ExyAJkUT9jGBJC6eoLq8liAJEMDkAIqnU-M2gUQtfuJUXsAovY-MWofQNSC3JsFAMMyZnM",
      "continuous":false,
      "database":null,
      "doc_id":null,
      "doc_write_failures":0,
      "docs_read":4,
      "docs_written":4,
      "missing_revisions_found":4,
      "replication_id":"1aebc296b560ed25c8153cda8f55069e+create_target",
      "revisions_checked":4,
      "source":"http://217.12.247.99:5984/account%2fa4%2f87%2f97afcbf4c1782d55613a5f86b0bf-201710/",
      "source_seq":"4-g1AAAAG5eJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfYgdRnMiUy5QB67pYGhYWKiGTY9-ExyAJkUT9jGBJC6eqiNjGAbzSwtUw1SDEm0MY8FSDI0ACmgYfPx2wpRuwCidj_CZsMkI0NLM1OybD4AMe0-MTY_gKgFhU0WAEpFdiQ",
      "started_on":1516776027,
      "target":"http://217.12.247.98:5984/account%2fa4%2f87%2f97afcbf4c1782d55613a5f86b0bf-201710/",
      "through_seq":"1-g1AAAAF1eJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfYgdRnMiYy5QB67pYGhYWKiGTY9-ExyAJkUT9jGBJC6eoLq8liAJEMDkAIqnU-M2gUQtfuJUXsAovY-MWofQNSC3JsFAMMyZnM",
      "type":"replication",
      "updated_on":1516794240,
      "user":null
   },
   {  
      "node":"couchdb@127.0.0.1",
      "pid":"<0.7273.48>",
      "changes_pending":234248,
      "checkpoint_interval":30000,
      "checkpointed_source_seq":"2041-g1AAAAJveJyd0EsKwjAQANBoBQVXbtzWE0iTfpKs7EEEzZdSqq5c60106S30Bh5Bdx4jpknBTRFaAhOYYR4zUwEAJkUgwUwcjqKQPIcILyP7YGVLQwZ4aIwpi4B9djYxpipiCtO2hj8MX9jIV17iAAymDuNE6BSRrlheY5sfFjpMYqkFk12xbY2dmh2Nl5REgneV9iMbwdl-Frs0o80dyGgidQZ7gVcP3htw7Q-XERHrfuDDg69635vTMJEsiVUv7e01d72n0zSkBKW8ra_8AuJDpH8",
      "continuous":false,
      "database":null,
      "doc_id":null,
      "doc_write_failures":0,
      "docs_read":2542,
      "docs_written":2542,
      "missing_revisions_found":2542,
      "replication_id":"260c79c446f46814cdf361b5d02b59ce+create_target",
      "revisions_checked":2542,
      "source":"http://217.12.247.99:5984/account%2f63%2ff8%2f14795fd5b8c74aa04ee865c52f73-201711/",
      "source_seq":"2541-g1AAAAJ4eJyd0LsNwjAQBmBDkKClpggsgGLnYbsirEET_FQUBaioYRPoGAM2gRWYIDh2UBqElOik_6Q73VdcCQCY5J4EU3E4ilzyFCK8DEzB0qyGDHC_qqoi9zgAg9nOzMZUBUxh-uvmj8TnJvmqxTYW40ToGJGuWFpjWYtlFpNYasFkV2xbY6cWWztMSSR4V2w_MgnOphnv0oALCzIaSZ3AXuDVgfcGvLnfJUSEuh_4cODTgOxtNUwki0LVS3s57ftA34IaUoJi_uu0-AAzHaN5",
      "started_on":1516774615,
      "target":"http://217.12.247.98:5984/account%2f63%2ff8%2f14795fd5b8c74aa04ee865c52f73-201711/",
      "through_seq":"2541-g1AAAAJ4eJyd0LsNwjAQBmBDkKClpggsgGLnYbsirEET_FQUBaioYRPoGAM2gRWYIDh2UBqElOik_6Q73VdcCQCY5J4EU3E4ilzyFCK8DEzB0qyGDHC_qqoi9zgAg9nOzMZUBUxh-uvmj8TnJvmqxTYW40ToGJGuWFpjWYtlFpNYasFkV2xbY6cWWztMSSR4V2w_MgnOphnv0oALCzIaSZ3AXuDVgfcGvLnfJUSEuh_4cODTgOxtNUwki0LVS3s57ftA34IaUoJi_uu0-AAzHaN5",
      "type":"replication",
      "updated_on":1516792740,
      "user":null
   }
]

@sergey-safarov
Copy link
Author

I guess that issue related to databases that contains docs with attachments.

@nickva
Copy link
Contributor

nickva commented Jan 24, 2018

Thanks for the report, Sergey

You might be encountering this issue: #745

A few questions to help narrow down the cause:

  • What is your exact version of CouchDB, is it 2.0.0, 2.1.1 or other

  • What is the size distribution of your documents (average, what are the largest ones)

  • What are the attachment sizes, what is their distribution (average, largest)

  • Are the databases which failed to replicate enough the ones with attachments? Do you have any that don't have attachments and those replicated fast enough?

  • If you can attach the contents of these config sections (but sanitize them to not contain database paths, username and other such things): couchdb, replicator, chttpd and cluster. It's easy to get those via the node local port (5986 one) and the _config section:

Here is an example of my config from the replicator section (jq part is the json pretty printer)

curl -s http://<adminuser>:<adminpas>@localhost:15986/_config/replicator | jq '.'
{
  "startup_jitter": "5000",
  "connection_timeout": "30000",
  "max_churn": "20",
  "interval": "60000",
  "retries_per_request": "5",
  "max_jobs": "500",
  "verify_ssl_certificates": "false",
  "http_connections": "20",
  "ssl_certificate_max_depth": "3",
  "socket_options": "[{keepalive, true}, {nodelay, false}]",
  "worker_processes": "4",
  "worker_batch_size": "500"
}

@nickva
Copy link
Contributor

nickva commented Jan 24, 2018

@sergey-safarov

See update to related ticket #745 (comment)

Try that fix and see if it changes anything.

Thanks again for your report

@sergey-safarov
Copy link
Author

Hello @nickva
I use RPM package http://apache.bintray.com/couchdb-rpm/el7/x86_64/couchdb-2.1.1-2.el7.centos.x86_64.rpm
Average document size with attacmetns - 2,3Gb / 12876 docs = 179 byte per doc
Largest doc - not more 1 Kb without attachments
Largest attachment - 2,4Mb
Average attachment - 800Kb
Think all databases without attachments is replicated normally.
Formated config is attached (used default).

{  
   "native_query_servers":{  
      "query":"{mango_native_proc, start_link, []}"
   },
   "httpd_design_handlers":{  
      "_compact":"{couch_mrview_http, handle_compact_req}",
      "_info":"{couch_mrview_http, handle_info_req}",
      "_list":"{couch_mrview_show, handle_view_list_req}",
      "_rewrite":"{couch_httpd_rewrite, handle_rewrite_req}",
      "_show":"{couch_mrview_show, handle_doc_show_req}",
      "_update":"{couch_mrview_show, handle_doc_update_req}",
      "_view":"{couch_mrview_http, handle_view_req}",
      "_view_changes":"{couch_mrview_http, handle_view_changes_req}"
   },
   "uuids":{  
      "algorithm":"sequential",
      "max_count":"1000"
   },
   "cluster":{  
      "n":"3",
      "q":"8"
   },
   "cors":{  
      "credentials":"false"
   },
   "chttpd":{  
      "backlog":"512",
      "bind_address":"0.0.0.0",
      "docroot":"./share/www",
      "port":"5984",
      "prefer_minimal":"Cache-Control, Content-Length, Content-Range, Content-Type, ETag, Server, Transfer-Encoding, Vary",
      "require_valid_user":"false",
      "socket_options":"[{recbuf, 262144}, {sndbuf, 262144}, {nodelay, true}]"
   },
   "httpd_global_handlers":{  
      "/":"{couch_httpd_misc_handlers, handle_welcome_req, <<\"Welcome\">>}",
      "_active_tasks":"{couch_httpd_misc_handlers, handle_task_status_req}",
      "_all_dbs":"{couch_httpd_misc_handlers, handle_all_dbs_req}",
      "_config":"{couch_httpd_misc_handlers, handle_config_req}",
      "_plugins":"{couch_plugins_httpd, handle_req}",
      "_replicate":"{couch_replicator_httpd, handle_req}",
      "_restart":"{couch_httpd_misc_handlers, handle_restart_req}",
      "_session":"{couch_httpd_auth, handle_session_req}",
      "_stats":"{couch_stats_httpd, handle_stats_req}",
      "_system":"{chttpd_misc, handle_system_req}",
      "_utils":"{couch_httpd_misc_handlers, handle_utils_dir_req, \"./share/www\"}",
      "_uuids":"{couch_httpd_misc_handlers, handle_uuids_req}",
      "favicon.ico":"{couch_httpd_misc_handlers, handle_favicon_req, \"./share/www\"}"
   },
   "attachments":{  
      "compressible_types":"text/*, application/javascript, application/json, application/xml",
      "compression_level":"8"
   },
   "query_server_config":{  
      "os_process_limit":"100",
      "reduce_limit":"true"
   },
   "vendor":{  
      "name":"The Apache Software Foundation"
   },
   "replicator":{  
      "connection_timeout":"30000",
      "http_connections":"20",
      "interval":"60000",
      "max_churn":"20",
      "max_jobs":"500",
      "retries_per_request":"5",
      "socket_options":"[{keepalive, true}, {nodelay, false}]",
      "ssl_certificate_max_depth":"3",
      "startup_jitter":"5000",
      "verify_ssl_certificates":"false",
      "worker_batch_size":"500",
      "worker_processes":"4"
   },
   "ssl":{  
      "port":"6984"
   },
   "log":{  
      "file":"/var/log/couchdb/couchdb.log",
      "level":"info",
      "writer":"file"
   },
   "indexers":{  
      "couch_mrview":"true"
   },
   "view_compaction":{  
      "keyvalue_buffer_size":"2097152"
   },
   "query_servers":{  
      "coffeescript":"./bin/couchjs ./share/server/main-coffee.js",
      "javascript":"./bin/couchjs ./share/server/main.js"
   },
   "features":{  
      "scheduler":"true"
   },
   "daemons":{  
      "auth_cache":"{couch_auth_cache, start_link, []}",
      "compaction_daemon":"{couch_compaction_daemon, start_link, []}",
      "external_manager":"{couch_external_manager, start_link, []}",
      "httpd":"{couch_httpd, start_link, []}",
      "index_server":"{couch_index_server, start_link, []}",
      "os_daemons":"{couch_os_daemons, start_link, []}",
      "query_servers":"{couch_proc_manager, start_link, []}",
      "uuids":"{couch_uuids, start, []}",
      "vhosts":"{couch_httpd_vhost, start_link, []}"
   },
   "couch_peruser":{  
      "delete_dbs":"false",
      "enable":"false"
   },
   "httpd":{  
      "allow_jsonp":"false",
      "authentication_handlers":"{couch_httpd_auth, cookie_authentication_handler}, {couch_httpd_auth, default_authentication_handler}",
      "bind_address":"0.0.0.0",
      "default_handler":"{couch_httpd_db, handle_request}",
      "enable_cors":"false",
      "enable_xframe_options":"false",
      "max_http_request_size":"67108864",
      "port":"5986",
      "secure_rewrites":"true",
      "socket_options":"[{recbuf, 262144}, {sndbuf, 262144}]",
      "vhost_global_handlers":"_utils, _uuids, _session, _users"
   },
   "httpd_db_handlers":{  
      "_all_docs":"{couch_mrview_http, handle_all_docs_req}",
      "_changes":"{couch_httpd_db, handle_db_changes_req}",
      "_compact":"{couch_httpd_db, handle_compact_req}",
      "_design":"{couch_httpd_db, handle_design_req}",
      "_design_docs":"{couch_mrview_http, handle_design_docs_req}",
      "_local_docs":"{couch_mrview_http, handle_local_docs_req}",
      "_temp_view":"{couch_mrview_http, handle_temp_view_req}",
      "_view_cleanup":"{couch_mrview_http, handle_cleanup_req}"
   },
   "database_compaction":{  
      "checkpoint_after":"5242880",
      "doc_buffer_size":"524288"
   },
   "csp":{  
      "enable":"true"
   },
   "couch_httpd_auth":{  
      "allow_persistent_cookies":"false",
      "auth_cache_size":"50",
      "authentication_db":"_users",
      "authentication_redirect":"/_utils/session.html",
      "iterations":"10",
      "require_valid_user":"false",
      "timeout":"600"
   },
   "couchdb":{  
      "attachment_stream_buffer_size":"4096",
      "changes_doc_ids_optimization_threshold":"100",
      "database_dir":"./data",
      "default_security":"admin_local",
      "delayed_commits":"false",
      "file_compression":"snappy",
      "max_dbs_open":"500",
      "os_process_timeout":"5000",
      "uuid":"aa251fc49c416b4b37f8ba8b1eb6f1aa",
      "view_index_dir":"./data"
   },
   "compactions":{  
      "_default":"[{db_fragmentation, \"70%\"}, {view_fragmentation, \"60%\"}]"
   },
   "compaction_daemon":{  
      "check_interval":"300",
      "min_file_size":"131072"
   }
}

Next i will test recommended fix

@sergey-safarov
Copy link
Author

I increased max_http_request_size up to 10 times - issue is still persist.
I can create VM with installed couchdb master and one database that cannot be replicated.
If you provide your public ssh key, then you can look issue on host.

@wohali
Copy link
Member

wohali commented Feb 1, 2018

@sergey-safarov please follow #745, this is likely a known issue with replication of documents with attachments.

@wohali wohali changed the title Very slow replication for some of databases Very slow replication for databases with attachments Feb 1, 2018
@wohali wohali added the bug label Feb 1, 2018
@janl janl added this to the 2.2.0 milestone Mar 5, 2018
@wohali
Copy link
Member

wohali commented Mar 27, 2018

@sergey-safarov have you tried the latest master to see if this is resolved? Multiple patches have landed since 25 January.

@janl janl removed this from the 2.2.0 milestone Mar 27, 2018
@janl janl closed this as completed Jul 14, 2018
@sergey-safarov
Copy link
Author

sergey-safarov commented Nov 21, 2018 •
edited
Loading

This page may be key for understanding of slow performance of containerized app
https://sysdig.com/blog/container-isolation-gone-wrong/

@sergey-safarov sergey-safarov mentioned this issue Nov 21, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug waiting on user
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@janl @wohali @nickva @sergey-safarov