数据库先检查再插入还是直接插入再处理异常,哪个更快?

Posted by Keal on October 18, 2023

起因是因为某天发现grafana上面, 服务有一段时间响应时间却异常变慢但请求量不高的情况,排查到最后发现那段时间数据库有大量的主键异常错误. 当然不是说这是主要原因,而是我想探究下这两种处理方式的区别.

测试条件

  • 数据库: postgres: 9.6.20
  • 脚本: python+sqlalchemy

测试流程

两种方式分辨测试按总数据容量和重复数据容量两个数据维度,各三个数据维度,一共3种组合来测试

测试代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
import time
import uuid
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker, scoped_session
import json

session = scoped_session(sessionmaker())
session_kdm = scoped_session(sessionmaker())
raw_data = {
    "kdm_data": "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n<DCinemaSecurityMessage xmlns=\"http://www.smpte-ra.org/schemas/430-3/2006/ETM\" xmlns:dsig=\"http://www.w3.org/2000/09/xmldsig#\" xmlns:enc=\"http://www.w3.org/2001/04/xmlenc#\">\n  <!-- Generated by Deluxe (1.2.7) -->\n  <AuthenticatedPublic Id=\"ID_AuthenticatedPublic\">\n    <MessageId>urn:uuid:7ffbbc10-ff8c-4014-9be5-400a523f5384</MessageId>\n    <MessageType>http://www.smpte-ra.org/430-1/2006/KDM#kdm-key-type</MessageType>\n    <AnnotationText>MiracleClub_FTR-3_S_EN-EN-OCAP_UK-IE_51-VI_4K_LION_20231005_DLX_IOP_VF</AnnotationText>\n    <IssueDate>2023-10-13T08:46:49+00:00</IssueDate>\n    <Signer>\n      <dsig:X509IssuerName>dnQualifier=4Pzb9BEbYscBBpJLUuniHXqNDvc=,CN=.dlx-1.intermediate.2,OU=dc,O=bydeluxe.com</dsig:X509IssuerName>\n      <dsig:X509SerialNumber>7775888885095907842</dsig:X509SerialNumber>\n    </Signer>\n    <RequiredExtensions>\n      <KDMRequiredExtensions xmlns=\"http://www.smpte-ra.org/schemas/430-1/2006/KDM\">\n        <Recipient>\n          <X509IssuerSerial>\n            <dsig:X509IssuerName>dnQualifier=4dl0oY64k/gzxFwgTB0eISmnFhg=,CN=.Cinea.MFGCA.1,O=DC256.Cinea.Com,OU=MFGCA1.DC256.Cinea.Com</dsig:X509IssuerName>\n            <dsig:X509SerialNumber>41493</dsig:X509SerialNumber>\n          </X509IssuerSerial>\n          <X509SubjectName>dnQualifier=6mpLoE014qD1faSNu4rt2TqbZCw=,CN=SM LE.Dolby256-CAT862-0007cf6a,O=DC256.Cinea.Com,OU=DolbyMediaBlock</X509SubjectName>\n        </Recipient>\n        <CompositionPlaylistId>urn:uuid:4f1ca430-93e0-4084-a422-e0f3a330f448</CompositionPlaylistId>\n        <ContentTitleText>MiracleClub_FTR-3_S_EN-EN-OCAP_UK-IE_51-VI_4K_LION_20231005_DLX_IOP_VF</ContentTitleText>\n        <ContentKeysNotValidBefore>2023-10-12T23:01:00+00:00</ContentKeysNotValidBefore>\n        <ContentKeysNotValidAfter>2024-01-04T03:59:00+00:00</ContentKeysNotValidAfter>\n        <AuthorizedDeviceInfo>\n          <DeviceListIdentifier>urn:uuid:76233f77-aa2b-4cb4-b43d-e525e4dc554c</DeviceListIdentifier>\n          <DeviceList>\n            <CertificateThumbprint>2jmj7l5rSw0yVb/vlWAYkK/YBwk=</CertificateThumbprint>\n          </DeviceList>\n        </AuthorizedDeviceInfo>\n        <KeyIdList>\n          <TypedKeyId>\n            <KeyType>MDIK</KeyType>\n            <KeyId>urn:uuid:083e3a2e-4163-41db-9afa-23b33419bb4b</KeyId>\n          </TypedKeyId>\n          <TypedKeyId>\n            <KeyType>MDAK</KeyType>\n            <KeyId>urn:uuid:30b090f4-ad63-4186-9892-3209e7810f02</KeyId>\n          </TypedKeyId>\n          <TypedKeyId>\n            <KeyType>MDIK</KeyType>\n            <KeyId>urn:uuid:21392ec7-f8fc-47ab-8063-28d97ca63903</KeyId>\n          </TypedKeyId>\n          <TypedKeyId>\n            <KeyType>MDAK</KeyType>\n            <KeyId>urn:uuid:36e88490-811d-49de-8923-d0fb441c6052</KeyId>\n          </TypedKeyId>\n          <TypedKeyId>\n            <KeyType>MDIK</KeyType>\n            <KeyId>urn:uuid:c96163e2-7dd3-4cbb-9cc6-ab701342fd9a</KeyId>\n          </TypedKeyId>\n          <TypedKeyId>\n            <KeyType>MDAK</KeyType>\n            <KeyId>urn:uuid:4656881c-8bab-4894-a177-41a695b3666b</KeyId>\n          </TypedKeyId>\n          <TypedKeyId>\n            <KeyType>MDIK</KeyType>\n            <KeyId>urn:uuid:506d0808-a42b-43a3-b761-380f7e175740</KeyId>\n          </TypedKeyId>\n          <TypedKeyId>\n            <KeyType>MDAK</KeyType>\n            <KeyId>urn:uuid:7035cbd6-32b4-4500-b1d2-e8421384e08d</KeyId>\n          </TypedKeyId>\n          <TypedKeyId>\n            <KeyType>MDIK</KeyType>\n            <KeyId>urn:uuid:d2dd13d4-5f7e-49a9-88af-eae752bf397e</KeyId>\n          </TypedKeyId>\n          <TypedKeyId>\n            <KeyType>MDAK</KeyType>\n            <KeyId>urn:uuid:a800e307-acf3-456e-93fe-684d7142d34c</KeyId>\n          </TypedKeyId>\n          <TypedKeyId>\n            <KeyType>MDIK</KeyType>\n            <KeyId>urn:uuid:84c6f4b1-a95e-40d6-8c8f-e9c44407e9e4</KeyId>\n          </TypedKeyId>\n          <TypedKeyId>\n            <KeyType>MDAK</KeyType>\n            <KeyId>urn:uuid:0fa9ceea-1de5-43fa-ac08-c9b3d6db544d</KeyId>\n          </TypedKeyId>\n        </KeyIdList>\n      </KDMRequiredExtensions>\n    </RequiredExtensions>\n    <NonCriticalExtensions/>\n  </AuthenticatedPublic>\n  <AuthenticatedPrivate Id=\"ID_AuthenticatedPrivate\">\n    <enc:EncryptedKey xmlns:enc=\"http://www.w3.org/2001/04/xmlenc#\">\n      <enc:EncryptionMethod Algorithm=\"http://www.w3.org/2001/04/xmlenc#rsa-oaep-mgf1p\">\n        <ds:DigestMethod xmlns:ds=\"http://www.w3.org/2000/09/xmldsig#\" Algorithm=\"http://www.w3.org/2000/09/xmldsig#sha1\"/>\n      </enc:EncryptionMethod>\n      <enc:CipherData>\n        <enc:CipherValue>Zp5PTLWSRs2G/RC2TQRbQM3noQVuMI029s631Kc+3gLAdjpLSAIGG4jkCRu4y47AN38ybHdxYHIC\nOOoj7W58esmQwlcEJhlXBnNpmSeTIzb2/SqH+dfsHBRxSeTW4PulwTfIC1iab+U0QkSxoDcQfcsP\n6tQrOOmVDYcYrBfKr7vigfXlXMtP/AaxdCRqEZoVRNygRYrYroP0VixSujUMdX/BMVuxT9GJid55\ntJLOvT6k5RLOoSjs+kN69lBl5+MWDCLMolwSryZXpA7fKuMlLBhNQ6zrY645vOh8AXISeEH/vYL3\nkH/Vq4cdRBcAuDToWJTiFYnTjZPR5UMA7jVnnQ==</enc:CipherValue>\n      </enc:CipherData>\n    </enc:EncryptedKey>\n    <enc:EncryptedKey xmlns:enc=\"http://www.w3.org/2001/04/xmlenc#\">\n      <enc:EncryptionMethod Algorithm=\"http://www.w3.org/2001/04/xmlenc#rsa-oaep-mgf1p\">\n        <ds:DigestMethod xmlns:ds=\"http://www.w3.org/2000/09/xmldsig#\" Algorithm=\"http://www.w3.org/2000/09/xmldsig#sha1\"/>\n      </enc:EncryptionMethod>\n      <enc:CipherData>\n        <enc:CipherValue>WThF1+8UIqbwpeO5gfIIvvehSVCIomGQ/07ws0Nfdc+nyhhcChqxu79Oa7LuO2rjeC/fkttqD2B6\n+Ghd9DR41LRe5oIpwJbzH6jADGCIqPHZcThbQfey13mxW2Ey2DWsOyzlvcr6iLVBjgsGFKa8b5nN\nssuZmXPYmrEdvK5yZYRUgbwo4QpjXJ3EMOYuRSE16Bi3ck39ViammoTlfwRZSCR/IwrJ1NoUMFnZ\nzCoInOjfdfTCdvvhIls0azqFvnt/BPV10gc3yDFxkzcdejetUIsM1qSQyDXEzqiHPxxwpBU8mq1J\nfLBlErF0xDy5J++/8HcNFcIYT3SXBV08v7eGCg==</enc:CipherValue>\n      </enc:CipherData>\n    </enc:EncryptedKey>\n    <enc:EncryptedKey xmlns:enc=\"http://www.w3.org/2001/04/xmlenc#\">\n      <enc:EncryptionMethod Algorithm=\"http://www.w3.org/2001/04/xmlenc#rsa-oaep-mgf1p\">\n        <ds:DigestMethod xmlns:ds=\"http://www.w3.org/2000/09/xmldsig#\" Algorithm=\"http://www.w3.org/2000/09/xmldsig#sha1\"/>\n      </enc:EncryptionMethod>\n      <enc:CipherData>\n        <enc:CipherValue>ESFQHkejmiSc7H9loXVDPQeNmExUt18WxKd9zyZxPBu7iDeYLhd807dTn2GhBTo52mB46L0m/mKI\nq41tUXF4c+g5y/Z0Eo8HcLDTSM+VEk0Y13PTWDemC9qcqgoVFfa1yoyj2/On6+NV3EY36zygdVI/\n38DE/HMOmao1A3D49D8KtUBGWqCqqeYmj1SLCajSIrPy2UUTAmn0fo5XbxPOn6og04Spf41JE6RY\nW1zPDVP42RTw4hPLwRA33W6yZdRKC5srzr5qYuJ/gP/SKLXZoMgNklvfhuxIOJFrso98a8dGEn03\nkrqkx6TJod3MGrreC8NER2M1fnMUGxJSHw5lWA==</enc:CipherValue>\n      </enc:CipherData>\n    </enc:EncryptedKey>\n    <enc:EncryptedKey xmlns:enc=\"http://www.w3.org/2001/04/xmlenc#\">\n      <enc:EncryptionMethod Algorithm=\"http://www.w3.org/2001/04/xmlenc#rsa-oaep-mgf1p\">\n        <ds:DigestMethod xmlns:ds=\"http://www.w3.org/2000/09/xmldsig#\" Algorithm=\"http://www.w3.org/2000/09/xmldsig#sha1\"/>\n      </enc:EncryptionMethod>\n      <enc:CipherData>\n        <enc:CipherValue>qKgUL9uyZI8TSIFYev9C7oyByLSyokOjOtKww2Ebn/ioTq5BwlvpS1LzbPcpNsl4+ENTXsqnynda\nbnVCGkU3DB6LQbDiXR5SVhxdJwO/eXyF+YzixVPBjhNYN6gR/Pq8D75Ndfh7VamgkLlRgUM8WYUL\nqizL+ZECE8ikscXWdeIIoZLmDJb4rcqyRPPvNtjIfDHuqgG/cLQ/oHtZUma+e/TggZIpT70Ko/hY\nZeSi/Kt0d8IJg8nQK4ubemD3PjGbD/aBtsIgH0xRn53TB8m/7GnaCAvgzFWylcyniAqmr4bioXno\nSVEIoocYGOpmU6L/dPhL5spwBPFbsUK4ul1jYQ==</enc:CipherValue>\n      </enc:CipherData>\n    </enc:EncryptedKey>\n    <enc:EncryptedKey xmlns:enc=\"http://www.w3.org/2001/04/xmlenc#\">\n      <enc:EncryptionMethod Algorithm=\"http://www.w3.org/2001/04/xmlenc#rsa-oaep-mgf1p\">\n        <ds:DigestMethod xmlns:ds=\"http://www.w3.org/2000/09/xmldsig#\" Algorithm=\"http://www.w3.org/2000/09/xmldsig#sha1\"/>\n      </enc:EncryptionMethod>\n      <enc:CipherData>\n        <enc:CipherValue>rcpCt9ipl6DY4SmsLgo+f8qR+0qxAT/WZVIIeALfKRHSGe7+11Z0FW5nU4uz8qDfReQbzCTZkGtV\npsQTnIJLDuFWsR/LXZ7m794RuhK9X9nK8omNTRl988hWW5rOeuO3PPGwsyTLEHXiNNCD3ZcwvqJl\nmJg2oMhpfGJwCb3R4NY08yE8MzjMMukEgZlKZJ58bXpiQGP3FT++1qLES/Q1wLDaVMUF4rSyOX8o\nJNhmNRbkVqCVvyY6HJZgPJyonlAvAgb2JdIJDkovoMj08sLcq1zsWVgC608mNkZdBtQzEFI0xMP/\nkuhSpUizPl2XDz2IlV65ruAubz14hbGi4TzdGg==</enc:CipherValue>\n      </enc:CipherData>\n    </enc:EncryptedKey>\n    <enc:EncryptedKey xmlns:enc=\"http://www.w3.org/2001/04/xmlenc#\">\n      <enc:EncryptionMethod Algorithm=\"http://www.w3.org/2001/04/xmlenc#rsa-oaep-mgf1p\">\n        <ds:DigestMethod xmlns:ds=\"http://www.w3.org/2000/09/xmldsig#\" Algorithm=\"http://www.w3.org/2000/09/xmldsig#sha1\"/>\n      </enc:EncryptionMethod>\n      <enc:CipherData>\n        <enc:CipherValue>EubQMfQxtT/3G52vedX234hv9sGBbP3YfQ0q6dBf1MPXZ7qKcvrRhKglk0ebDIY4wvfPdSb9nl6n\noIeYxIMvWkiQUMMiT8hTjpH/Pf2o2AkRmFBYp+QzuL6boVE/+MlWN8G/oNbB59Eo4oZrq4m61b2/\nJWQU9hc/FwstI+ET6+eglUkmNwHT+/rrXg5FAUxZi0QsKDc7r9jDvRA+aX+u8HYvAIVIiQEKhRsL\ntI4Vr3cpsHFNw0wc12ZO75Y+YeB308b0IH4axNu87ibdd0uKlbSE8AO4sJeDTaJiFxsyGFC7OFCN\n8gs6zkO5TLP0+3lC7BUKE/m8ExdVcpcXb854cQ==</enc:CipherValue>\n      </enc:CipherData>\n    </enc:EncryptedKey>\n    <enc:EncryptedKey xmlns:enc=\"http://www.w3.org/2001/04/xmlenc#\">\n      <enc:EncryptionMethod Algorithm=\"http://www.w3.org/2001/04/xmlenc#rsa-oaep-mgf1p\">\n        <ds:DigestMethod xmlns:ds=\"http://www.w3.org/2000/09/xmldsig#\" Algorithm=\"http://www.w3.org/2000/09/xmldsig#sha1\"/>\n      </enc:EncryptionMethod>\n      <enc:CipherData>\n        <enc:CipherValue>ZGYUwKYNtOfAd+0vpxCPKCw9jELsH/Fr9KoCxkwLxin+R22ngkJKBFCGNiYFFz7Mp58sUiqbM6qR\n1ywL2CAvBLeeUrn6K5QLyZtXYtl/7IWRiXK8fzOCKyTUIwmTYf2oYgmYRDpAvrq2DPu8fSOIDnZP\nrKbv6E2lv7Zf3Ge4nxeHt9rL7qSWzmjS91P++mEplwtLswP3azB6lUqF4PmqNaoAKeHla0CrVFAo\niP5zE4Vup1crBh1q+d/c5tJMsqRUZIKbfrCX+xwWH9t9lCdpJkR4GNnYSzTgq2ByH+v7n5nYf44c\nQ1XRHUpEoa91w+v02yydUfOIY9BGoFO3dGPJCQ==</enc:CipherValue>\n      </enc:CipherData>\n    </enc:EncryptedKey>\n    <enc:EncryptedKey xmlns:enc=\"http://www.w3.org/2001/04/xmlenc#\">\n      <enc:EncryptionMethod Algorithm=\"http://www.w3.org/2001/04/xmlenc#rsa-oaep-mgf1p\">\n        <ds:DigestMethod xmlns:ds=\"http://www.w3.org/2000/09/xmldsig#\" Algorithm=\"http://www.w3.org/2000/09/xmldsig#sha1\"/>\n      </enc:EncryptionMethod>\n      <enc:CipherData>\n        <enc:CipherValue>OYyyiP6csd/vPI8Kyd/XahO3Lf2iyKAkb5RQgR197LKUsBNHZ3Au6rZhQIz+r99SNai48WJ0gXfo\nuAsP/2fMovtROJHZHFeEqkYywOa4ejindd80L948f71dAZrwgTAbYtHwUIKs6cg7lB2W4GtShMFT\nSQALVCsXpH4sUSvPHmQ8co46jVGshFqPRcjpgZxrekskN3693uxYXsDEo58Twfq54Sfz42YSAEB3\nhz1LwDi7yMzUEa0YUOu86yAq/IpIpSmymLVYHcns02Owo/Mbg94WjOqOzkBjROSFlydufODFYPp9\nh9vf3wcuyIjfjl8FO41nF1JndU+kUhBjGqpoYQ==</enc:CipherValue>\n      </enc:CipherData>\n    </enc:EncryptedKey>\n    <enc:EncryptedKey xmlns:enc=\"http://www.w3.org/2001/04/xmlenc#\">\n      <enc:EncryptionMethod Algorithm=\"http://www.w3.org/2001/04/xmlenc#rsa-oaep-mgf1p\">\n        <ds:DigestMethod xmlns:ds=\"http://www.w3.org/2000/09/xmldsig#\" Algorithm=\"http://www.w3.org/2000/09/xmldsig#sha1\"/>\n      </enc:EncryptionMethod>\n      <enc:CipherData>\n        <enc:CipherValue>WAsSZ2wF6S43fqUYQsy505WKiUzkz9rCYaQ84tvadROKieIZNJ4nPcmplikcWQmFH0EJk6Ap6RgK\nTrVL9cWZ1szQZIEcJpW1Ipod5LgJSgDSO9QbSMErp+hkgYkJlPraJh6qmw15tomqh5JaeLYEzG9Q\nX2mG7R2D/HXpuiLWWIQfSRwl/Z+TWFVJlQMVKA6qr0bHcONvme3Am5io3fcd3N6F996aXpb8dc/0\nJPGV/DzpbmRaCAweujNMc6Hv3LPPfc29+5+nHh+V0LDi7krqxvOTk/wXTc2GLWlnKRQOEG/7Awdh\nw3phQbJHFDjpwkQ3Nz0Wvyti4rxGkjCrVh9H6A==</enc:CipherValue>\n      </enc:CipherData>\n    </enc:EncryptedKey>\n    <enc:EncryptedKey xmlns:enc=\"http://www.w3.org/2001/04/xmlenc#\">\n      <enc:EncryptionMethod Algorithm=\"http://www.w3.org/2001/04/xmlenc#rsa-oaep-mgf1p\">\n        <ds:DigestMethod xmlns:ds=\"http://www.w3.org/2000/09/xmldsig#\" Algorithm=\"http://www.w3.org/2000/09/xmldsig#sha1\"/>\n      </enc:EncryptionMethod>\n      <enc:CipherData>\n        <enc:CipherValue>kJ+Qup92N67cHVZwZ4uFV+B7u9E8rTBhq8D6ojT8aXrNs8G7eYLJhN3W1xYGBKSV+myEX1BNfT9e\n28YufI4nedMvVn04J+yTzxT7ih77sXwfwY+fctcDii7nfvOk63OMoIEOGH+HpZs9ojwRkWsHnjv6\nsFED8HsjTgkTBGB5Qt8lj7Th1hbz7zrLe7P8MsHvDQyeu1I00ZK/N0ArxOYu5HTBWC2aFRvk+OQz\nYT+zCMzHdXDxaSQNXy+C9cec3WFZE0ZcW1IsBqzEG4HuSUssFJY1vA89b3PWCTk8KE/dFMZfQ7//\nNsf68dQl25eXFltABYHGCXLMOegCFQooQTPn/Q==</enc:CipherValue>\n      </enc:CipherData>\n    </enc:EncryptedKey>\n    <enc:EncryptedKey xmlns:enc=\"http://www.w3.org/2001/04/xmlenc#\">\n      <enc:EncryptionMethod Algorithm=\"http://www.w3.org/2001/04/xmlenc#rsa-oaep-mgf1p\">\n        <ds:DigestMethod xmlns:ds=\"http://www.w3.org/2000/09/xmldsig#\" Algorithm=\"http://www.w3.org/2000/09/xmldsig#sha1\"/>\n      </enc:EncryptionMethod>\n      <enc:CipherData>\n        <enc:CipherValue>OXmi+LeX88jP51OmcBhrHn6nzXyNuSt0CuH655fYarFStHYD1FTBmmmyaD0QaZ2BD70FJgNA0zXx\nMaj+QZyzqo14PhvTH0UsZQSDU1AjPna4qiO39QggcJL4vqs3fsQYjJWSOhSRSXIuQ0TpaPLU/GsN\nFO81U6UCEfXA8IpqA93aLVos/4GS4SH2vpAWFbRMk7NaxffzRv8jscBwO8ZI4CxDuUmAZwnaw2bI\nSc+5biaT+GWWuZ6MiurwtbMMuTm8uE99ItSKE0SNmBMOfVcT3kVyQy9HMzVn1SGVdiCV6+DFK8Zy\nXEJmaneE/3cKC5HaxBuDMIhYsWtaraNPkGBZvw==</enc:CipherValue>\n      </enc:CipherData>\n    </enc:EncryptedKey>\n    <enc:EncryptedKey xmlns:enc=\"http://www.w3.org/2001/04/xmlenc#\">\n      <enc:EncryptionMethod Algorithm=\"http://www.w3.org/2001/04/xmlenc#rsa-oaep-mgf1p\">\n        <ds:DigestMethod xmlns:ds=\"http://www.w3.org/2000/09/xmldsig#\" Algorithm=\"http://www.w3.org/2000/09/xmldsig#sha1\"/>\n      </enc:EncryptionMethod>\n      <enc:CipherData>\n        <enc:CipherValue>jtQnk2+ld5AtUwOdrEjcSqybpMikRpO5QiWcRww8qInqaxEEAuFHjS9y8klSJsV18gb2WNYQJwIU\nQqpnGfOL+P5qj+fVqZtHM/ODhjdqNlr9c2tXw/00elxi0LtDjC5PweIg+knMHnRXChyN2a0kSBwT\nCQj4JQKtAlpcF/qNX+FAige/bmyalSnSZfFd5IbIuJw62lmzfuaau0A/Yi5GnG+DQU5PiWQ72bTm\nK9OtV39nXaagcLftKsIbd1xKC2kKh4xcR+B4efSHkwSVymrSOih2d8SSD44FjuAirmR3JRkNM/3v\niilpeTsE16Y6BmFx+l7iFgAopvK7IG2/9K0VDg==</enc:CipherValue>\n      </enc:CipherData>\n    </enc:EncryptedKey>\n  </AuthenticatedPrivate>\n  <dsig:Signature xmlns:dsig=\"http://www.w3.org/2000/09/xmldsig#\">\n    <dsig:SignedInfo>\n      <dsig:CanonicalizationMethod Algorithm=\"http://www.w3.org/TR/2001/REC-xml-c14n-20010315#WithComments\"/>\n      <dsig:SignatureMethod Algorithm=\"http://www.w3.org/2001/04/xmldsig-more#rsa-sha256\"/>\n      <dsig:Reference URI=\"#ID_AuthenticatedPublic\">\n        <dsig:DigestMethod Algorithm=\"http://www.w3.org/2001/04/xmlenc#sha256\"/>\n        <dsig:DigestValue>Lvy/q+0F33lZ0fge1bJvbU+1E6OiYMOwN4FW5+1q9co=</dsig:DigestValue>\n      </dsig:Reference>\n      <dsig:Reference URI=\"#ID_AuthenticatedPrivate\">\n        <dsig:DigestMethod Algorithm=\"http://www.w3.org/2001/04/xmlenc#sha256\"/>\n        <dsig:DigestValue>mVTNklh/glPsaGW5AlEowG6mMiZACzMC3oiYd3IVg+Q=</dsig:DigestValue>\n      </dsig:Reference>\n    </dsig:SignedInfo>\n    <dsig:SignatureValue>1yYdIYrq69m0kGL8xT95p1ylrfcAj+OM8+V6DeJekJi6jzg9KC4nczHwYRGT5qMxpyReyNymhetx\nyHCFAPNwFrusuRUfqwZoYoe5ZqX3tM1J8aLJKfy1CXr0WUiTDLf6dvLbc+OLHrzftsHKJ+IntYvV\ndwNp1II0Vgfo7dxJSGq6L+ILCP9jNtlpf7AYqpBRG+wv7oIRBRPvRDCeJQXcJLN5+wjbOjeuGzHr\nKJWYXSFDizn0g2W0I/S6m8LQjAvEKxkXMt7uyw6CMVJUOD4kTaRcBmXmSFSzUPsdQga8yAZRTJZB\n21MUXUCCqAdTZZetw+medJBs28JwXiZXKh3e+A==</dsig:SignatureValue>\n    <dsig:KeyInfo>\n      <dsig:X509Data>\n        <dsig:X509IssuerSerial>\n          <dsig:X509IssuerName>dnQualifier=4Pzb9BEbYscBBpJLUuniHXqNDvc=,CN=.dlx-1.intermediate.2,OU=dc,O=bydeluxe.com</dsig:X509IssuerName>\n          <dsig:X509SerialNumber>7775888885095907842</dsig:X509SerialNumber>\n        </dsig:X509IssuerSerial>\n        <dsig:X509Certificate>MIIEJjCCAw6gAwIBAgIIa+mBvhkMwgIwDQYJKoZIhvcNAQELBQAwazEVMBMGA1UE\nChMMYnlkZWx1eGUuY29tMQswCQYDVQQLEwJkYzEeMBwGA1UEAxMVLmRseC0xLmlu\ndGVybWVkaWF0ZS4yMSUwIwYDVQQuExw0UHpiOUJFYllzY0JCcEpMVXVuaUhYcU5E\ndmM9MB4XDTIxMDQxOTIyNDMyN1oXDTM2MDQxNjIyNDMyN1owYzEVMBMGA1UEChMM\nYnlkZWx1eGUuY29tMQswCQYDVQQLEwJkYzEWMBQGA1UEAxMNQ1MuZGx4LTEubGVh\nZjElMCMGA1UELhMcWXhxQ1ljZTg3K3dvM0YwL0IxSlVub2xnb3E4PTCCASIwDQYJ\nKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOL9evKWKa5jyfgiUYj6KWxeLa+wO8qx\nj4J1p4MSQBdHcoKKEWW2+V7BwV16cLih8GrGcot60biMsuRKmuUHtUqH7Oljqepy\n0OeL6now7Cg+gBlx5xhHFmu9yl7/FjTBBNy3NdODWuBeIX7+LR50QgrWOyWmz43v\nijgJqUIATMj+gggK+TGtSv5RxL6g1fwjPRdIa14RpLz3gKBMPLAlh66DRZRqssgh\ntr4FLVdqmipA0Id2D2hO9CXvGp+gsU0U+4QQU0w5MBD3QcBt0iAfvfa5n54QTTJW\nNr+Q8qctg5yut8jz40Qopo3spMowW77IOvZguIXDsxaDfH6AVG/7B+0CAwEAAaOB\n1TCB0jAMBgNVHRMBAf8EAjAAMAsGA1UdDwQEAwIFoDAdBgNVHQ4EFgQUYxqCYce8\n7+wo3F0/B1JUnolgoq8wgZUGA1UdIwSBjTCBioAU4Pzb9BEbYscBBpJLUuniHXqN\nDvehb6RtMGsxFTATBgNVBAoTDGJ5ZGVsdXhlLmNvbTELMAkGA1UECxMCZGMxHjAc\nBgNVBAMTFS5kbHgtMS5pbnRlcm1lZGlhdGUuMTElMCMGA1UELhMca3N5Z1NIY1Fz\nVWVRWW96dktTOEFjUXFuNjVFPYIBCjANBgkqhkiG9w0BAQsFAAOCAQEAgHDGbz8H\ndYWBvmnC5egLZVoTgmiXAZZmPJukHAUjypP6eihz2RmV5LJsO9wPfQB03veh+PRP\n3GTHPo+raLPrX8vTioN2IoUhGMF1ntu90vhJrseWVBWauXrqC1uEtZBDi05cppNr\n9E6P3kafnBP+9YLvHQEp3zMr943GI//wHkBR3r9r4rMLDRbver1HZlKi6/7l8Cug\nP2c4t5gmu/h9Ri5jD8eXwoCDitnpe4TeRzgsLrFVIPOijejaP7WeGeSLSiG8wHp8\nQMfmPIUEviPaMiV36ErsNoc0+1YYKPtOxi39E7UYJH1GrHG3Mily3yitJAT0sTTY\no6pP/tmpvheLJQ==</dsig:X509Certificate>\n      </dsig:X509Data>\n      <dsig:X509Data>\n        <dsig:X509IssuerSerial>\n          <dsig:X509IssuerName>dnQualifier=ksygSHcQsUeQYozvKS8AcQqn65E=,CN=.dlx-1.intermediate.1,OU=dc,O=bydeluxe.com</dsig:X509IssuerName>\n          <dsig:X509SerialNumber>10</dsig:X509SerialNumber>\n        </dsig:X509IssuerSerial>\n        <dsig:X509Certificate>MIIEIzCCAwugAwIBAgIBCjANBgkqhkiG9w0BAQsFADBrMRUwEwYDVQQKEwxieWRl\nbHV4ZS5jb20xCzAJBgNVBAsTAmRjMR4wHAYDVQQDExUuZGx4LTEuaW50ZXJtZWRp\nYXRlLjExJTAjBgNVBC4THGtzeWdTSGNRc1VlUVlvenZLUzhBY1FxbjY1RT0wHhcN\nMjEwNDE5MjI0MzI2WhcNMzYwNDE3MjI0MzI2WjBrMRUwEwYDVQQKEwxieWRlbHV4\nZS5jb20xCzAJBgNVBAsTAmRjMR4wHAYDVQQDExUuZGx4LTEuaW50ZXJtZWRpYXRl\nLjIxJTAjBgNVBC4THDRQemI5QkViWXNjQkJwSkxVdW5pSFhxTkR2Yz0wggEiMA0G\nCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC539avbxJjLNlPC5jCosLoLbJOMDiF\npYSwBh0QU8tTX/4X8tRTejfJnGSZ8heKnCbmxPMIXWNBCqQd4P++phb6DsEI1NG/\nIpeMmr7elm2sTpK1xiKoG614qzXQfpvASzWd5G4URu8HANDDTLQPJPs7HtcNKaov\n2Y7k0KXXGSFV9tNkvDYmOHWZBFL2m56c9nqzZchaD12/uZXShNYk2VyLooEe3EiQ\njbhcWEEbQL87MftSdmWsUrm7ti7Mv71u0DM+8/26Z5r1EFW/DsInRegLuotVdVaQ\nzKH+lhMMJK2d5v/svA+48wLTmE/URmTPE76AuaICmeaUK3AJ8K9fH/71AgMBAAGj\ngdEwgc4wEgYDVR0TAQH/BAgwBgEB/wIBAjALBgNVHQ8EBAMCAQYwHQYDVR0OBBYE\nFOD82/QRG2LHAQaSS1Lp4h16jQ73MIGLBgNVHSMEgYMwgYCAFJLMoEh3ELFHkGKM\n7ykvAHEKp+uRoWWkYzBhMRUwEwYDVQQKEwxieWRlbHV4ZS5jb20xCzAJBgNVBAsT\nAmRjMRQwEgYDVQQDEwsuZGx4LTEucm9vdDElMCMGA1UELhMcckhDKzJMd0lkMXlt\nKzNZMGZrbld3T2hUbEwwPYIBMzANBgkqhkiG9w0BAQsFAAOCAQEAUIbgOIc1P+TE\neQOH5L3CaV2HZ8HgtaLVSMylKOsln9/ymR3Djs4BAPhJU6PvU/PU0YtVvpBUaCYP\nzFE0CSd1QR8xv5lgVmThnElcJ42m85qAnGUBZ+APjKPK7aHG69tDzr5w+kz9Inye\n6kQThpbT04fNa9xxvh1vFQ7n00peZ69rZvxscZq/s2J2mNvtE4CzgyE/2LZFilu/\njdLqN3te19biVIeGiVkhACE3wGTcb6aZgrBPRnb1iHildEf5VzOEfYg7kb54adNM\nhdr7HtTwI/SOkHHM8ye1VUcj0zSofAuIOEjCLr5dJCZIgApWjoJUbD+NVFNKNhIi\nkbkTAe4zyQ==</dsig:X509Certificate>\n      </dsig:X509Data>\n      <dsig:X509Data>\n        <dsig:X509IssuerSerial>\n          <dsig:X509IssuerName>dnQualifier=rHC\\+2LwId1ym\\+3Y0fknWwOhTlL0=,CN=.dlx-1.root,OU=dc,O=bydeluxe.com</dsig:X509IssuerName>\n          <dsig:X509SerialNumber>51</dsig:X509SerialNumber>\n        </dsig:X509IssuerSerial>\n        <dsig:X509Certificate>MIIEGjCCAwKgAwIBAgIBMzANBgkqhkiG9w0BAQsFADBhMRUwEwYDVQQKEwxieWRl\nbHV4ZS5jb20xCzAJBgNVBAsTAmRjMRQwEgYDVQQDEwsuZGx4LTEucm9vdDElMCMG\nA1UELhMcckhDKzJMd0lkMXltKzNZMGZrbld3T2hUbEwwPTAeFw0yMTA0MTkyMjQz\nMjZaFw0zNjA0MTgyMjQzMjZaMGsxFTATBgNVBAoTDGJ5ZGVsdXhlLmNvbTELMAkG\nA1UECxMCZGMxHjAcBgNVBAMTFS5kbHgtMS5pbnRlcm1lZGlhdGUuMTElMCMGA1UE\nLhMca3N5Z1NIY1FzVWVRWW96dktTOEFjUXFuNjVFPTCCASIwDQYJKoZIhvcNAQEB\nBQADggEPADCCAQoCggEBAPWWIIYdsVZIzfJrgwtxyVhutDb9Y9jXaBRcaTU/ilUV\ngTAz5n+LmtCxMStR8o4v7DY/W241UMWwzS/2wXhmJV45CMyxvF+kdMjYz+6eX8SJ\n++X/tJAqh8aqnipahC1egVVod8QE6ixK+r7wjZrm3yxtVnHrMPO3YYiFt3jIsQGT\n0lJkJnKBhBf6JLbQTiMw3hWbBLiDvckUvbpjQTom4w6DbgY6ogkUQ30LzlHYGdim\nZs2N60Md0HWHktQR02jM0MXsUiGfQBnOg6oTbqml2TAvG7Js2cRVfuDgpis28p5H\nVXAc/xGI5lyAY4VUH6wkW5q4xyoN+6Utx8oLs1EpX2cCAwEAAaOB0jCBzzASBgNV\nHRMBAf8ECDAGAQH/AgECMAsGA1UdDwQEAwIBBjAdBgNVHQ4EFgQUksygSHcQsUeQ\nYozvKS8AcQqn65EwgYwGA1UdIwSBhDCBgYAUrHC+2LwId1ym+3Y0fknWwOhTlL2h\nZaRjMGExFTATBgNVBAoTDGJ5ZGVsdXhlLmNvbTELMAkGA1UECxMCZGMxFDASBgNV\nBAMTCy5kbHgtMS5yb290MSUwIwYDVQQuExxySEMrMkx3SWQxeW0rM1kwZmtuV3dP\naFRsTDA9ggIA1DANBgkqhkiG9w0BAQsFAAOCAQEAwia7hXcC5poVI1AxbIi9NHY9\nH1oYmtjPC7fBDm4pZUZrD/f33ev8QDlka9t7gZj97UQZXLQRF7cW5O/iKELE/jd5\neLJmHtE+KJ6ol0TlIs5bWhKNR8ArpoID7kfQ09R6dzKaFkMOtKiBxF9oKQ/L0Bhj\nVZLLHK54rH/5rOSdYnUa4UkWZnpdgs+PeiMMTPeoZDy2SpAPT15E1/SIce/pqKEv\n4ZmUroAi63SwftVgH5oFu0CvZDHu7kcs1Fa+s33JtTAUfq92ynhFgroQY/mJle8v\nHA0UDKKYLQ4xHBQFy7TysQzbk8ILd9lub8Qa5Bs5aWPeoYFrrgF1gWlDJqDjBg==</dsig:X509Certificate>\n      </dsig:X509Data>\n      <dsig:X509Data>\n        <dsig:X509IssuerSerial>\n          <dsig:X509IssuerName>dnQualifier=rHC\\+2LwId1ym\\+3Y0fknWwOhTlL0=,CN=.dlx-1.root,OU=dc,O=bydeluxe.com</dsig:X509IssuerName>\n          <dsig:X509SerialNumber>212</dsig:X509SerialNumber>\n        </dsig:X509IssuerSerial>\n        <dsig:X509Certificate>MIIEETCCAvmgAwIBAgICANQwDQYJKoZIhvcNAQELBQAwYTEVMBMGA1UEChMMYnlk\nZWx1eGUuY29tMQswCQYDVQQLEwJkYzEUMBIGA1UEAxMLLmRseC0xLnJvb3QxJTAj\nBgNVBC4THHJIQysyTHdJZDF5bSszWTBma25Xd09oVGxMMD0wHhcNMjEwNDE5MjI0\nMzI2WhcNMzYwNDE5MjI0MzI2WjBhMRUwEwYDVQQKEwxieWRlbHV4ZS5jb20xCzAJ\nBgNVBAsTAmRjMRQwEgYDVQQDEwsuZGx4LTEucm9vdDElMCMGA1UELhMcckhDKzJM\nd0lkMXltKzNZMGZrbld3T2hUbEwwPTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC\nAQoCggEBANRJSqz2cHvBkj0qKCn89ZaCdlzBBcYf1OqXOH16LikF4/8HhSOQIBFB\niCqmGe1uvGvUdyv1PilW6f0RmZx3MN7+0waN4FlARGLsjz3kpCImkS6AW1mYCsj8\nr1f/sk0pA2gL6z7CL42PKZOXLTRtoGWqhKiWdIbeMEc/WCfqbWnFjBg7I5QcSAki\n+na7zDtsGwkH5y+rP+IEcOmF6DR4NhNoLuEuFRTxDVPXd/KVUQqBSGmoQ+fX0Flw\nVE8cHiZJ49WT1VQGjT1r6cK7MPYB9KIdoTLNE0X+vk2RwJb/ZMEdBrEX99RTdgM/\nRl7ow0zjhoyn9hQXIcNiaIXHy49nlkkCAwEAAaOB0jCBzzASBgNVHRMBAf8ECDAG\nAQH/AgEDMAsGA1UdDwQEAwIBBjAdBgNVHQ4EFgQUrHC+2LwId1ym+3Y0fknWwOhT\nlL0wgYwGA1UdIwSBhDCBgYAUrHC+2LwId1ym+3Y0fknWwOhTlL2hZaRjMGExFTAT\nBgNVBAoTDGJ5ZGVsdXhlLmNvbTELMAkGA1UECxMCZGMxFDASBgNVBAMTCy5kbHgt\nMS5yb290MSUwIwYDVQQuExxySEMrMkx3SWQxeW0rM1kwZmtuV3dPaFRsTDA9ggIA\n1DANBgkqhkiG9w0BAQsFAAOCAQEAeYRe0tRCim8WdtUbVB7IQIcGLqXjodfz5pKo\n8RSicw1DM0Ty2Jn2nuHBwCKKnCg6VfZkhxkrJad7nrn3lXtK4rxgPAXwLPTZBIAQ\n8Bvgs9UeBKZn+5T9NwnLtge2Hn9e4U0BVUk2tJK+bmmMegiQmLdTpeOH8q+ksuCU\nH++gI6fu6UT6COM3TyKmRvebpC/Mq2K5bDK7bQhFZXc4Qm5MsYNhQE5uaiaLpAx+\nKiLdWz7Q3Ja2Dq1rWS4jIcww91O/nFVtDiebfsOE35zSDMx9CejowOVshrIlyhzZ\n2PCdSSWYAddZvEKqFeLQZD4CBTKTCdxMx5d6YK3T2YGX21YuMg==</dsig:X509Certificate>\n      </dsig:X509Data>\n    </dsig:KeyInfo>\n  </dsig:Signature>\n</DCinemaSecurityMessage>"}
data = json.dumps(raw_data)


def calculate_execution_time(func):
    def wrapper(*args, **kwargs):
        start_time = time.time()
        result = func(*args, **kwargs)
        end_time = time.time()
        execution_time = end_time - start_time
        print(f"函数 {func.__name__} 执行时间为: {execution_time} 秒")
        return result

    return wrapper


def init_dev_db():
    username = 'postgres'
    password = 'postgres'
    host = 'localhost'
    name = 'server'
    db_url = f'postgresql://{username}:{password}@{host}:35432/{name}'
    engine = create_engine(db_url, pool_pre_ping=True, echo=False, pool_recycle=3600)
    session.configure(bind=engine)


@calculate_execution_time
def insert_message_queue_direct(data_num, repeat_num):
    """循环执行sql插入数据,利用数据库唯一索引异常来处理重复键"""

    def _insert(uid):
        sql = f"""INSERT INTO message_queue (uuid, agent_uuid, msg_type, msg_body, delay_until, priority) VALUES ('{uid}'::uuid, '64417d32-9d93-5346-9568-7ebefa8b4887'::uuid, 'kdm_delivery', '{data}', NULL, 50)"""
        try:
            session.execute(sql)
            session.commit()
        except Exception as e:
            # logging.exception(e)
            session.rollback()

    uuids = [str(uuid.uuid4()) for i in range(data_num)]

    for i in range(data_num):
        _insert(uuids[i])

    repeat_num = data_num if repeat_num > data_num else repeat_num
    for i in range(repeat_num):
        _insert(uuids[i])
    print(f'data_num:{}')


@calculate_execution_time
def insert_message_queue_after_check(data_num, repeat_num):
    """循环执行sql插入数据,提前查询处理重复键"""

    def _insert(uid):
        query_sql = f"""select uuid from message_queue where uuid='{uid}'"""
        query_res = session.execute(query_sql).fetchall()
        if query_res:
            pass
            # logging.warning(f'uuid {uuid} already exists')
        else:
            sql = f"""INSERT INTO message_queue (uuid, agent_uuid, msg_type, msg_body, delay_until, priority) VALUES ('{uid}'::uuid, '64417d32-9d93-5346-9568-7ebefa8b4887'::uuid, 'kdm_delivery', '{data}', NULL, 50)"""
            session.execute(sql)
            session.commit()

    uuids = [str(uuid.uuid4()) for i in range(data_num)]

    for i in range(data_num):
        _insert(uuids[i])

    repeat_num = data_num if repeat_num > data_num else repeat_num
    for i in range(repeat_num):
        _insert(uuids[i])


if __name__ == '__main__':
    init_dev_db()
    data_scope = [(5000, 0), (10000, 5000), (50000, 10000)]
    for data_num, repeat_num in data_scope:
        insert_message_queue_direct(data_num, repeat_num)
        insert_message_queue_after_check(data_num, repeat_num)

结果

第一次测试

数据集(总数据/重复数据) 直接插入耗时(s) 先检查在插入(s)
5000,0 46.37 57.01
10000,5000 141.57 103.89
50000,10000 542.00 530.55

第二次测试

提前清空了表

数据集(总数据/重复数据) 直接插入耗时(s) 先检查在插入(s)
10000,0 80.80 97.84
10000,2000 108.77 103.80
10000,4000 139.122 99.99
10000,6000 120.06 94.39
10000,8000 134.06 114.10

第三次测试

数据集(总数据/重复数据) 直接插入耗时(s) 先检查在插入(s)
10000,1000 123.34 92.06
10000,3000 109.14 118.27
10000,5000 153.03 105.28
10000,7000 148.18 129.42
10000,9000 181.22 120.72

第四次测试,相比第三次,这次插入数据的大小缩小了几乎100倍.

数据集(总数据/重复数据) 直接插入耗时(s) 先检查在插入(s)
10000,1000 101.60 69.64
10000,3000 59.54 68.64
10000,5000 71.32 68.77
10000,7000 95.42 99.73
10000,9000 117.36 93.95

总结

  1. 重复数据越多,先检测在插入效率越高. 反之重复数据越少,直接插入效率越高
  2. 插入数据中,大概有10%的重复数据时,两者性能差不多.
  3. 插入的数据越大,提前检测的优势会越明显.