That's a good question. Starting with, how exactly is the package version determined? If we assume stricter header layout for v6 then it can't really be in a header tag because we need to know the format before parsing it. A new header magic I guess, but that feels heavy-handed when the header format is for all practical purposes still the same. But then, it may be well worth it simply for the clean break -effect.

The file order in payload doesn't match the one in the header, at least for hardlinks and other stuff that may not have content at all. Maybe in a perfect world one could expect that order to be strictly defined and always correct, but there are multiple cases in rpm history where one bug or the other has led into order changing or things like %ghost payload in the header where it's not expected. So I for one will sleep a lot better knowing there's a mapping from payload to the header.

The file order in payload doesn't match the one in the header, at least for hardlinks and other stuff that may not have content at all. Maybe in a perfect world one could expect that order to be strictly defined and always correct, but there are multiple cases in rpm history where one bug or the other has led into order changing or things like %ghost payload in the header where it's not expected. So I for one will sleep a lot better knowing there's a mapping from payload to the header.

The point would be to not allow arbitrary ordering anymore. If if the payload is a raw data area, there is no way to deviate from the header. So instead of allowing to refer back and forth between payload and header we only allow pointing from the header into the payload. The rules are rather simple:

• the order of file data is the same order as those of the file related arrays
• offset and length of the data for each file are calculated based based on the FILESIZES array
• skip non regular files (FILEMODES)
• skip ghost files aka the marker that the payload does not contain the file's data (FILEFLAGS)
• each unique inode only once aka don't duplicate hardlinked data (FILEINODES)

If at the same time we restrict FILEINODES to numbers from 0 to max nfiles-1, then no extra inode search/lookup would be needed either.

Yep, of course such a format is possible to design. Just saying that this is not in our plans. The scope of change in the v6 format is intentionally quite restricted to avoid it becoming a forever project.

GH ticket, PR and discussions activity are gated to rpm-maint (eg start of this thread is here, and AFAIK replying via email does work. We've no intention to have a separate discussion on any mailing list, but those not willing to be on GH for any reason should be able to participate nevertheless.

No problem. I just didn't know if I should subscribe to a list like the old rpm list. I can track this ticket.

You’re welcome. What about padding between one tag and the next?

Personally, I think that using a dribble for the padding is a fine, provided that:

1. No other tags are allowed in the dribble.
2. Regions are checked to be consistent.
3. The padding is required to be in the dribble.
4. The padding is ignored by hdrblobImport().

That said, the current solution is simpler and much more obvious from the perspective of package reading.

What about padding between one tag and the next?

See above, "you'll have an explicit requirement for all padding to be zeros in v6" (emphasis added)

That said, the current solution is simpler and much more obvious from the perspective of package reading.

In some ways yeah, but the padding is an annoying wrinkle that requires separate functions/special cases to read what is supposed to be just two headers in a row. So a self-padding signature header (whatever the exact means) would be a nice simplification for the reader.

Whether dribbles or not etc, I totally agree there needs to be exactly one accepted way.

What does "dribble" mean in this context?

Relevant #2462

FWIW, dribbles are now (briefly) documented in https://rpm-software-management.github.io/rpm/manual/format_header.html#immutable-regions

I'm all for properly supporting multi-arch in packages in principle (the multiarch deps are even mentioned in the v6 spec here), but getting rid of "arch" is likely to be, uh, complicated. Even if rpm itself could, the repodata and every single depsolver out there has very deeply wired assumptions about that. Very unlikely to happen in this timeframe.

What about adding per-file architectures, and keeping the legacy "arch" purely for compatibility with old tools?

Add more of the stuff that we want to get rid of? 😄

Well, RPM has the "fat" arch for multi-arch RPMs already... So detecting arches per file and using "fat" as the rpmarch would work for me.

Actually, maybe we should rename fat to multi instead and use that to inform RPM that it needs to process per-file architecture information. A new rpmarch would allow us to handle specifying the behavior for "universal binary" type RPMs, which are currently undefined.

I said that mtimes for files in the payload are the only thing that actually matters for clamping to SOURCE_DATE_EPOCH. We absolutely don't want SOURCE_DATE_EPOCH affecting package buildtimes by default.

Sorry for the miscommunication. The question I had started out with was

but the point of source_date_epoch is to make packages reproducable right? If you allow the buildtime to be different, then the package is no longer reproducable, and there's no reason to use source_date_epoch

You pointed out that the purpose of use_source_date_epoch_as_buildtime is that various infrastructure needs a meaningful, non-static buildtime to function properly, but that there is still value in having a reproducible payload even if the rest of the package is not bit-identical. In order to get payload reproducibility you need to clamp the mtimes of the files using SOURCE_DATE_EPOCH, because that metadata is baked into the CPIO. So you want to specify SOURCE_DATE_EPOCH but keep the buildtime independent, and that requires having a configuration option - use_source_date_epoch_as_buildtime.

My point is that if the RPMv6 payload format doesn't include that mtime metadata in the archive metadata, then you can have payload reproducibility all the time and don't need to set SOURCE_DATE_EPOCH to get it. Therefore, if you want full reproducibility you could specify SOURCE_DATE_EPOCH, and if you don't want it, then you can skip SOURCE_DATE_EPOCH entirely, and you still get the property of having a reproducible payload regardless. You can collapse a bunch of options into just "do I want a fully reproducible package, or not".

That would also bring what rpmbuild does into closer alignment with the SOURCE_DATE_EPOCH specification, which states that "Build processes MUST use this variable for embedded timestamps in place of the "current" date and time." If it's set, it's supposed to be used. That's not practical now for all of the above reasons, so rpmbuild doesn't follow it strictly. But in theory RPMv6 could eliminate the practical issues and allow this to be followed strictly.

Granted as this behavior has now been in use for a while, it may not be easy to remove or change by default. I'm just saying that in theory, it could be simplified.

The "new" payload format doesn't carry the per-file info in the payload like cpio does, but we of course still store the file mtime in the header. Those aren't going anywhere at all, and so none of the source-date-epoch stuff is affected or even related.

but we of course still store the file mtime in the header.

Right, I understand, no disagreement there. If you need a fully reproducible package, you need to set source-date-epoch.

But if you don't need or even want a fully reproducible package (but you still want the payload to be reproducible), you no longer need to set source-date-epoch, whereas currently you do (and also a bunch of flags to disable aspects of it). It's all of those flags for disabling aspects of source-date-epoch which become potentially unnecessary.

@Conan-Kudo Is this making any sense?

See the very first comment in this topic from @DemiMarie - the format version should be signed to prevent changes. I suppose the lead could be signed, but I'm not sure we want to resurrect to repurpose this 25+ years old dead structure here.

Coming back to this: to avoid gratuitious compatibility breakage, we can't put either 0 or 6 in there.
The best we can do is put 4 in there because all of rpm 4.x used 3 as the major version for alleged LSB compatibility, so it at least differentiates it a bit. It's not such a terrible lie even because the fundamental header structure of v6 is basically v4...

Why can't 6 go there?

Because in rpm < 4.18, we have this silly check in there:

    if (lead->major < 3 || lead->major > 4) {
        *msg = xstrdup(_("unsupported RPM package version"));
        return RPMRC_FAIL;
    }

Putting 6 in the otherwise unused lead would render v6 packages unreadable by a huge range of older rpm versions that have otherwise zero problems accessing it, with precisely zero benefit to rpm. It's just not worth it.

An rpm cannot obviously carry a checksum on itself (for matching what's in the repodata). I think it's long overdue the repo metadata is overhauled to address the various defiencies in it, but that's another topic.

I don't see how rpm could help with repo metadata being non-extensible (despite initial intentions - it's quite ironical that the supposedly extensible xml became such an immovable obstacle).

As for changelogs, as you can see in this draft, no changes are planned.

What's the background on the issue you are referring to here?

If I am not mistaken - changelogs are stored in rpm header and they are uncompressed. Is there any reason to have them uncompressed?

Well, nothing in the header is compressed. Compressing the entire header would be more interesting (see #2220)

group tag made optional

Come to think of it, is this considered legacy nowadays? IIRC the Fedora packaging guidelines and other distros recommend against it, I believe comps kind of replaced it?

Group tag is still used by the Mandrake family (Mageia, OpenMandriva, PLD, ALT, etc.).

It may be considered legacy in Fedora but I disagree. It's not going anywhere.

Indeed, the choice of algorithms will need a second look. SHA-256 seems to be falling out of fashion already, and we'd like v6 to be future proof for at least a few years 😄

I wouldn't say SHA2-256 is falling out of fashion, most crypto-people seem to think there's not much risk of it being broken any time soon. But a bit of future proofing wouldn't hurt.

How would that make using the parallel arrays easier?

Oh, those... Thanks for the reminder. We'll indeed need to figure what to do with those because they are even indexes in the rpmdb. This is tracked as #2633 now.

There's no reason to link that to v6 in particular, additional data can be added any time. Head over to #2389 to discuss that.

See https://web.archive.org/web/20070621191805/https://www.redhat.com/archives/rpm-list/2001-April/msg00283.html

Whether it makes sense or not is another question.

Yup. The dependency is tracking the use of syntax that will create a package that won't work quite right with versions of rpm before 3.0.3.

...

Not a bug. The dependency is written with <= so that the range is closed, as >= would make the implicit promise "forever". Meanwhile, since the matching capability is Provides: rpmlib(VersionedDependencies) = 3.0.3-1 there's only a single point covered by the overlapping dependency ranges.

It makes some sense, but it's definitely counterintuitive. The first statement isn't entirely true because of the "or equals". The second statement almost doesn't matter, because rpm presents provides with the old versions anyway and only the "equals" portion of the range is used. And if RPM presents them as a list of capabilities, then why not just match on the capabilities rather than dragging version numbers into it?

I guess to make error messages more informative? Assuming that is taken advantage of. If it's treated as a normal dependency I expect the resulting error message would be backwards.

File-based self-dependencies are not going anywhere in general, because rpm uses that info for its own purposes. Besides, they serve as a dependency generation sanity check, and avoid surprises when (not if) packages get split to smaller pieces.

Whether those %config() deps in particular serve any purpose is a whole other question. AIUI the alleged original use-case is to allow installing packages with --noconfigs while supplying alternative configuration via eg site-specific packages. Only it never worked (without resorting to --nodeps and such) because --noconfigs is global.

%readme as a spec directive may be obsolete, but there's no reason to drop support for the virtual file attribute. On the contrary, if it was properly integrated with %doc it would be nice to 'rpm --show-readme mypackage' instead of having to chase around in /usr/share/doc.

How would that work exactly? Add a %doc(readme) to replace %readme? Files marked as such would then have both doc and readme flags, similar to %config(noreplace) setting both config and noreplace flags?

That's a spec syntax issue, it doesn't matter for package format.