The following is a dump of an IMAP conversation between Ximian Evolution 1.0.3 and uw-imapd 4.7 with David Harris's maildir packages, as packaged by Mitel (imap-4.7-1mdir10es). The other bit of involved software is Mutt 1.2.5.1, although I've also duplicated the mutt bug in Mutt 1.4.
I'm convinced that I've found a bug in Mutt and in Evolution, and I'm not sure whether or not I've found a bug in imapd. IMAP is a hairy protocol that I'm still becoming accustomed to speaking, so I'd like you to read over the dump and my comments and make sure that my deductions and observations make sense, and that I haven't missed anything.
If you can't find me (mendel on rhizomatic, unixnet, OPN) on IRC, please send me your comments at richl-imap@e-smith.com. Thanks!
This is bug 29304 in Ximian's bugzilla.
This is bug 1321 in Mutt's bug-tracking system.
* OK localhost IMAP4rev1 v12.264 server ready
B00000 CAPABILITY
* CAPABILITY IMAP4 IMAP4REV1 NAMESPACE IDLE SCAN SORT MAILBOX-REFERRALS LOGIN-REFERRALS AUTH=LOGIN THREAD=ORDEREDSUBJECT
B00000 OK CAPABILITY completed
B00001 LOGIN "richl" "fnord"
B00001 OK LOGIN completed
B00002 NAMESPACE
* NAMESPACE (("" "/")("#mhinbox" NIL)("#mh/" "/")) (("~" "/")) (("#shared/" "/")("#ftp/" "/")("#news." ".")("#public/" "/"))
B00002 OK NAMESPACE completed
B00003 LIST "" ""
B00003 OK LIST completed
B00004 LIST "" "*"
* LIST () "/" Mail
* LIST () "/" Mail/IN.escalate
* LIST () "/" Mail/IN.bugzilla
* LIST () "/" Mail/IN.cron
* LIST () "/" Mail/IN.security
* LIST () "/" Mail/IN.devinfo
* LIST () "/" Mail/IN.postmaster
* LIST () "/" Mail/IN.boards
* LIST () "/" Mail/IN.sysadmin
* LIST () "/" Trash
* LIST () "/" "Sent Items"
* LIST () "/" formletters/security-breach
* LIST (\NoInferiors) NIL INBOX
B00004 OK LIST completed
B00005 STATUS "INBOX" (UNSEEN)
* STATUS INBOX (UNSEEN 2)
B00005 OK STATUS completed
OK, nothing unusual up there. (No, that's not my password. :-) Let's look at a mailbox.
B00006 SELECT "Mail/IN.bugzilla" * 57 EXISTS * 0 RECENT * OK [UIDVALIDITY 1029955482] UID validity status * OK [UIDNEXT 1029976767] Predicted next UID * FLAGS (\Answered \Flagged \Deleted \Draft \Seen) * OK [PERMANENTFLAGS ()] Permanent flags * OK [UNSEEN 11] first unseen message in Mail/IN.bugzilla B00006 OK [READ-WRITE] SELECT completed
Here's where things start to get odd:
B00007 UID FETCH 1:46 (FLAGS) B00007 OK UID FETCH completed
Already showing signs of confusion. Evolution is asking for UIDs 1 to 46, but it doesn't know where UIDs start yet. However, it does know that there are 46 (= 57 (EXISTS) - 11 (UNSEEN)) seen messages, and IMAP4 guarantees that message numbers begin at one and increment by one with each message. Evolution almost certainly should have said FETCH, not UID FETCH.
B00008 UID FETCH 1:* (FLAGS RFC822.SIZE)
That will work -- Evolution is asking for UIDs from 1 to the maximum -- but it still looks like it thinks it's using message numbers. That almost looks like a "Drat, I don't know what's going on, I'll ask for everything" line. Here's the response:
* 1 FETCH (UID 1029955483 FLAGS (\Seen) RFC822.SIZE 2401) * 2 FETCH (UID 1029955484 FLAGS (\Seen) RFC822.SIZE 2935) * 3 FETCH (UID 1029955485 FLAGS (\Seen) RFC822.SIZE 3901) * 4 FETCH (UID 1029955486 FLAGS (\Seen) RFC822.SIZE 1962) * 5 FETCH (UID 1029955487 FLAGS (\Seen) RFC822.SIZE 2085) * 6 FETCH (UID 1029955488 FLAGS (\Seen) RFC822.SIZE 1843) * 7 FETCH (UID 1029955489 FLAGS (\Seen) RFC822.SIZE 1654) * 8 FETCH (UID 1029955490 FLAGS (\Seen) RFC822.SIZE 2109) * 9 FETCH (UID 1029955491 FLAGS (\Seen) RFC822.SIZE 2203) * 10 FETCH (UID 1029955492 FLAGS (\Seen) RFC822.SIZE 3339) * 13 FETCH (UID 1029955493 FLAGS (\Seen) RFC822.SIZE 5823)
Now there's some oddness from the imap server. Where's 11 and 12? IMAP4rev1 says that message number increments by one for each message.
There are 44 messages in the FETCH response. I tested the similar command FETCH 1:* (UID) and imapd returned 57 messages, where the missing ones had UID 0. Now, 0 isn't in the range 1:*, so it's probably correct in not listing them -- but that means more than one message has UID 0!
* 14 FETCH (UID 1029955494 FLAGS (\Seen) RFC822.SIZE 2384) * 15 FETCH (UID 1029955495 FLAGS (\Seen) RFC822.SIZE 6341) * 16 FETCH (UID 1029955496 FLAGS (\Seen) RFC822.SIZE 1711) * 17 FETCH (UID 1029955497 FLAGS (\Seen) RFC822.SIZE 3454) * 18 FETCH (UID 1029955498 FLAGS (\Seen) RFC822.SIZE 2131) * 25 FETCH (UID 1029955499 FLAGS (\Seen) RFC822.SIZE 3282) * 26 FETCH (UID 1029955500 FLAGS (\Seen) RFC822.SIZE 2153) * 27 FETCH (UID 1029955501 FLAGS (\Seen) RFC822.SIZE 2236) * 28 FETCH (UID 1029955502 FLAGS (\Seen) RFC822.SIZE 2451) * 33 FETCH (UID 1029955503 FLAGS (\Seen) RFC822.SIZE 2911) * 35 FETCH (UID 1029955504 FLAGS (\Seen) RFC822.SIZE 2742) * 36 FETCH (UID 1029955505 FLAGS (\Seen) RFC822.SIZE 3189) * 37 FETCH (UID 1029955506 FLAGS (\Seen) RFC822.SIZE 3457) * 38 FETCH (UID 1029955507 FLAGS (\Seen) RFC822.SIZE 3987) * 39 FETCH (UID 1029955508 FLAGS (\Seen) RFC822.SIZE 5258) * 40 FETCH (UID 1029955509 FLAGS (\Seen) RFC822.SIZE 2897) * 41 FETCH (UID 1029955510 FLAGS (\Seen) RFC822.SIZE 10405) * 42 FETCH (UID 1029955511 FLAGS (\Seen) RFC822.SIZE 15878) * 43 FETCH (UID 1029955512 FLAGS (\Seen) RFC822.SIZE 2946) * 44 FETCH (UID 1029955513 FLAGS (\Seen) RFC822.SIZE 2874) * 45 FETCH (UID 1029955514 FLAGS (\Seen) RFC822.SIZE 3906) * 46 FETCH (UID 1029955515 FLAGS () RFC822.SIZE 3233) * 47 FETCH (UID 1029955516 FLAGS (\Seen) RFC822.SIZE 3581) * 48 FETCH (UID 1029976569 FLAGS (\Seen) RFC822.SIZE 2717) * 49 FETCH (UID 1029976570 FLAGS (\Seen) RFC822.SIZE 53770) * 50 FETCH (UID 1029976571 FLAGS (\Seen) RFC822.SIZE 1923) * 51 FETCH (UID 1029976572 FLAGS () RFC822.SIZE 2030) * 52 FETCH (UID 1029976573 FLAGS () RFC822.SIZE 2908) * 53 FETCH (UID 1029976574 FLAGS () RFC822.SIZE 54875) * 54 FETCH (UID 1029976575 FLAGS () RFC822.SIZE 3654) * 55 FETCH (UID 1029976576 FLAGS () RFC822.SIZE 2647) * 56 FETCH (UID 1029976577 FLAGS () RFC822.SIZE 7358) * 57 FETCH (UID 1029976578 FLAGS () RFC822.SIZE 3507) B00008 OK UID FETCH completed B00009 UID FETCH 1029955483:1029955516,1029976569:1029976578 BODY.PEEK[HEADER]
Ok, now Evolution asks for UIDs. Note the UID skip. I don't know if that's important, but it matches the skip between message number 47 and 48 in the first FETCH. I don't think the headers are particularly interesting, and they contain information from our internal bug-tracking system, so I've only shown the first, last, and an important one in between, and have elided the Subject:.
* 1 FETCH (UID 1029955483 BODY[HEADER] {851}
Return-Path:
Delivered-To: richl-filter@allspice.nssg.mitel.com
Received: (qmail 17451 invoked by alias); 17 Jun 2002 16:21:24 -0000
Delivered-To: alias-localdelivery-richl-filter@e-smith.com
Received: (qmail 17444 invoked by uid 5111); 17 Jun 2002 16:21:23 -0000
Delivered-To: richl@allspice.nssg.mitel.com
Received: (qmail 17440 invoked by alias); 17 Jun 2002 16:21:23 -0000
Delivered-To: alias-localdelivery-richl@e-smith.com
Received: (qmail 17436 invoked by uid 100); 17 Jun 2002 16:21:23 -0000
Date: 17 Jun 2002 16:21:23 -0000
Message-ID: <20020617162123.17435.qmail@e-smith.com>
From: gordonr@e-smith.com
To:
Cc: alreid@e-smith.com, richl@e-smith.com
Reply-to: e-smith-bugzilla-daemon@e-smith.com
Lines: 40
)
[...]
* 11 FETCH (UID 0 BODY[HEADER] {817}
That's one of the elided messages I mentioned earlier, with UID 0. I don't understand why it was returned in response to a request for the range 1029955483:1029955516,1029976569:1029976578. One guess: the imapd is converting the range to message numbers, in which case 11 (UID 0) falls within the range of 1 (UID 1029955483) to 46 (UID 1029955516).
[...]
* 57 FETCH (UID 1029976578 BODY[HEADER] {1209}
Return-Path:
Delivered-To: richl-deliver-bugzilla@allspice.nssg.mitel.com
Received: (qmail 19007 invoked by alias); 21 Aug 2002 23:09:25 -0000
Delivered-To: alias-localdelivery-richl-deliver-bugzilla@e-smith.com
Received: (qmail 19003 invoked by uid 5111); 21 Aug 2002 23:09:25 -0000
Delivered-To: richl-filter@allspice.nssg.mitel.com
Received: (qmail 18988 invoked by alias); 21 Aug 2002 23:09:24 -0000
Delivered-To: alias-localdelivery-richl-filter@e-smith.com
Received: (qmail 18982 invoked by uid 5111); 21 Aug 2002 23:09:24 -0000
Delivered-To: richl@allspice.nssg.mitel.com
Received: (qmail 18978 invoked by alias); 21 Aug 2002 23:09:24 -0000
Delivered-To: alias-localdelivery-richl@e-smith.com
Received: (qmail 18972 invoked by uid 100); 21 Aug 2002 23:09:24 -0000
Date: 21 Aug 2002 23:09:24 -0000
Message-ID: <20020821230924.18971.qmail@e-smith.com>
From: charlieb@e-smith.com
To: charlieb@e-smith.com
Cc: richl@e-smith.com
Subject: foo
Reply-to: e-smith-bugzilla-daemon@e-smith.com
X-Spam-Status: No, hits=-0.6 required=5.0 tests=NO_REAL_NAME,DIFFERENT_REPLY_TO,FROM_AND_TO_SAME,AWL version=2.20
Lines: 63
)
B00009 OK UID FETCH completed
Other than the range problem above, that seemed to go fine. The client got back a little more than it expected. There's a correlation between message filename and UID; all the messages with UIDs have what Bernstein calls info appended, like1026743729.4510.allspice:2,Sand all of the messages with UID 0 have no info, like1025902462.22738.allspiceIt is illegal to move a message into cur/ without adding info.
Aha! A bit of experimenting, and I've discovered that that occurs when mutt displays a message in its index but never in its pager; since it's seen it, it moves it to cur, but since you haven't read it, it doesn't add info. I'm not sure if this is a GIGO problem from the imap server's point of view; mutt's only doing half its job, and it's probably legitimate (if undesirable) for imapd to break.
But then we're back to misbehaving Evolution:
B00010 UID FETCH 57 BODY.PEEK[] B00010 OK UID FETCH completed B00011 UID FETCH 58 BODY.PEEK[] B00011 OK UID FETCH completed
Evolution asks for messages that don't exist. We've never seen anything with UIDs of 57 or 58, but we have seen something with a message number of 57 (but not 58!). I'm certain that's a case of Evolution confusing FETCH and UID FETCH.
B00012 LOGOUT * BYE allspice.nssg.mitel.com IMAP4rev1 server terminating connection B00012 OK LOGOUT completed
* 10 FETCH (UID 1029955492 FLAGS (\Seen) RFC822.SIZE 3339) * 13 FETCH (UID 1029955493 FLAGS (\Seen) RFC822.SIZE 5823)MSNs number existing messages sequentially and when messages are sorted by MSN, their UIDs are strictly monotonically increasing. Among other things, this implies that that for any two messages, uid2-uid1 >= msn2-msn1. uw-imapd (or more properly, the maildir patch for it) is violating that. The MSNs tell the client there there are two messages between number 10 and 13; the UIDs tell the client there can't be any. [thanks Arnt Gulbrandsen]