PGP contacts

[link2xt]

JSON-RPC: reset_contact_encryption API is removed.
In deltachat-rpc-client API Contact.reset_encryption is removed.
"Group tracking plugin" in legacy Python API was removed because it relied on parsing email addresses from system messages with regexps.

Known multi-device (non-)issue: pinning a 1:1 chat from legacy client pins email chat via sync message.

Still missing, not all of this should be fixed before merging but at least CI should pass:

• Return an error when API user tries to add email contact to encrypted chats.
• Same for PGP-contacts, they should not be added to unencrypted chats.
• Fix AEAP tests.
• Chat-Group-Member-Fpr header.
• Ignore Chat-Group-ID if the message is not encrypted+signed.
• Complete Thunderbird tests.
• Tests for 1:1 chat assignment. If we have multiple PGP-contacts with the same email address, can assign outgoing messages without Autocrypt-Gossip to the most recent one or using In-Reply-To or References. If there are no PGP-contacts, the message can go to email contact or trash. EDIT: we have tests for incomplete message assignment, they pass. This is not comprehensive and does not use Message-IDs and References, but tying chat assignment to contact lookup is complicated.
• Migration. Started at [WIP] feat: Migration for PGP-contacts #6818
• Do not list email-contacts in the contact list by default. For email contacts maybe add a flag.
• Open issues for Thunderbird: test_prefer_encrypt_mutual_if_encrypted has TODOs for assigning to chat by issuer fingerprint. This should also work if the key is attached and no Autocrypt header is sent. Signed-only messages should probably go to PGP-chat without a padlock. Fixing this is out of scope of this PR. Signed-only messages to to email-contact. If Thunderbird is configured to send Autocrypt header, chatting should work, this is already tested.
• SHA-256 fingerprints (API to get SHA2-256 fingerprints for any keys rpgp/rpgp#531, wip: "imprint" fn rpgp/rpgp#541). If we can, we should use SHA-256 fingerprints for v4 keys instead of having SHA-1 fingerprints as the primary key. Invite links with SHA-1 should still be supported. Adding a column can be done later and we will likely need two columns anyway with standard fingerprint and SHA-256 to support openpgp4fpr and invite links, but Chat-Group-Member-Fpr format should be fixed before release.

[Hocuri]

Just 3 small things I was wondering while skimming over

[adbenitez]

a way to know if a chat is an encrypted chat is missing, currently there is only "isPgpContact" that can be used to mark contacts/1:1 chat but for unencrypted groups/threads of classic email non-pgp contacts there is no API to recognize such chats and also put the "classic email" marker as for classic email contacts

[Hocuri]

This adds up to 2 additional database calls to loading BasicChat, though I can't think of a good way to avoid it, so maybe it's fine. Or maybe it should only be added to FullChat, idk.

[Hocuri]

One idea might be to have a separate chattype, SingleUnencrypted.

[Hocuri]

Loading the self fingerprint everytime a contact is looked up, just to check whether it's ContactId::SELF, seems unnecessarily slow to me.

I wonder if we can simplify & speed up things by not using the kepairs table anymore, and instead storing the private and public key in configs. This would mean that it's not possible to have multiple self keys anymore, but I'm not sure there is anyone who is actually making use of the fact that DC theoretically supports using multiple self keys. It would also mean that we have to store the key as base64, not sure about the consequences there.

An alternative would be to store the self fingerprint in a OnceCell on Context, since we don't allow changing the primary key anyway.

But we can open an issue for this, and do it in a follow-up PR.

[Hocuri]

@link2xt agrees with the second alternative (caching fingerprint in memory).

[Hocuri]

Another day, another set of review comments

[Hocuri]

That's a lot of code to calculate the to and past ids, would be nice to extract a function here:

async fn get_to_and_past_contacts(
    context: &Context,
    mime_parser: &MimeMessage,
    incoming_origin: Origin,
) -> Result<(Vec<Option<ContactId>>, Vec<Option<ContactId>>)> {
    let chat_id = if let Some(grpid) = mime_parser.get_chat_group_id() {
    ...
        past_ids = add_or_lookup_contacts_by_address_list(
            context,
            &mime_parser.past_members,
            Origin::Hidden,
        )
        .await?;
    };
    Ok((to_ids, past_ids))
}

    let (to_ids, past_ids) =
        get_to_and_past_contacts(context, &mime_parser, incoming_origin).await?;

[Hocuri]

What about unencrypted emails that are partially downloaded? They should be assigned to the email contact, not to some PGP contact with this email address. Maybe there is some code for this already and I just haven't seen it, otherwise we need a check for whether Content-Type is multipart/encrypted.

[link2xt]

We don't know if partially downloaded email is encrypted or not, we don't even prefetch content-type. Messages are assigned to some PGP-contact. After downloading, they will go to email-contact.

[Hocuri]

But this is not about prefetch but about partial downloads? And for partial downloads, we fetch the full headers, which includes Content-Type:

const BODY_PARTIAL: &str = "(FLAGS RFC822.SIZE BODY.PEEK[HEADER])";

[Hocuri]

Ah, wait, this is both about prefetch and about actual partial downloading.

What about this:

• Rename is_partial_download parameter to find_pgp_contact_by_addr
• In receive_imf_inner(), replace is_partial_download.is_some() with (pseudocode) is_partial_download.is_some() && headers.get("Content-Type") == "multipart/encrypted"

[Hocuri]

@link2xt agrees if it's easy to do

[link2xt]

There is a problem with 1:1 chat assignment. If there is no Autocrypt-Gossip on outgoing message, then to_id gets converted to email contact. If there is also a Chat-Verified header, receive_imf fails with an error like DeltaChat: [accId=1] src/imap.rs:1456: receive_imf error: Non-PGP contact Contact#10 cannot be verified.. and the message does not show up at all. So we should assign by References or In-Reply-To and 1:1 chat before converting to_ids.

[nicodh]

With this branch Instant Onboarding with group invite link does not work any more.

At least with desktop when pasting a group invite code in
Add Profile => Create Profile => Use other server => Scan Invitation Code BackendRemote.rpc.checkQr(accountId, url) throws an error:
"failed to decode https://i.delta.chat/# QR code: failed to add or lookup contact for address ContactAddress("lhnjqt2jh@nine.testrun.org"): No self addr configured"

Scanning the invite code with an existing user works fine.

@Hocuri @link2xt