Miroslav Suchý

On the third day of DevConf, I was in the kids' corner all day, which I ran together with Petr Blaho. I didn’t have time for anything else that day.

In the corner we had the Ada & Zengemann book by Matthias and a bunch of stickers.

We played a lot of Ricochet Robot - I like this game because quite often the one who comes up with an unusual and unexpected solution wins. Plus, an almost unlimited number of people can play at the same time and they all play at the same time. No waiting for a teammate to make a turn. The rules are so simple that they blur the distinction between an adult and a small child.

Mr. and Mrs. Mašláňovi brought Lego Coding Express - for me, it was the first time I’ve seen it live. A great educational toy for kindergartens. The Lego was pretty durable and I can definitely recommend it for kindergartens. What pleasantly surprised me was that you can print an adapter with which you can connect the original tracks to the wooden tracks they sell for wooden trains. Great!

I had a Cubetto for the little kids there. I also brought Lego WeDo and Micro:bit. Both were used and I gave two Micro:bits to kids who were interested. Hopefully they’ll make some things with them that they’ll enjoy.

We had about twenty kids hanging around all day. And by noon there were six (or maybe more) kids there at the same time, each at a different age - that was pretty wild.

One of the popular things was an Useless Box.

While I was at DevConf, my wife was on a wellness retreat with her sister, and she kept me supplied with photos from the retreat. So I tried to keep up and took a few pictures too.

If you have kids and you want to try something similar, you can use my notes (in Czech only).

Today started with a Keynote: What if you could boot a container? where Dan Walsh, Stef Walter and Colin Walters talked about bootable containers. The demo was a bit choppy, but otherwise the keynote was funny and easy to understand.

Then I moved on to Brian Stinson’s talk on RHEL 11. I didn’t learn anything new (as I am deep down involved), but it was a nice affirmation.

I went to the “Fedora Hatch Meetup @DevConf.com” where we spent a good portion of it discussing the budget and hardware needed for secondary architectures and unusual expansion cards.

Then I met Marrhias Kirschner from the FSFE, who gave me a copy of his book Ada & Zangemann and asked if I could take it to tomorrow’s Kids' Corner, where unfortunately he can’t make it. The book is aimed at young children (5-8 years old) and is an inspiration why to play with HW and SW and why it is worth fighting for free SW. That sounds very pathetic, but in the book it is presented as a fairy tale and in very simple language.

Then I had some free time before the “Fedora RISC-V” talk started, so I stopped by the VUT students' booth on deep fake and took a quiz to see if I could tell in an audio recording if it was a deep fake. I ended up staying there for 45 minutes - so I missed the RISC-V lecture. I thought I could recognize about 30% of the samples, but I recognized about 81%. Students will use the result in their future work.

Afterwards I had some more nice chats in the atrium and at the end I dropped in for Martin’s lightning talk “My experience with teaching beginners”. Nothing new for me, but a nice affirmation.

In few minutes I am going to move to social event at Dobrák. I will not report from there because “What happen Las Vegas, stays in Las Vegas”.

The first thing that surprised me was that when I arrived at 9:00 (the keynote started at 9:30), the courtyard was already full of people. And I had some interesting conversations in the first half hour.

The opening keynote was boring.

Then I planned to go to the talk “Creating more meaningful Fedora release notes”, but the talk was cancelled at the last minute. So I moved on to the “Upstream Meetup”. Tomas and Lenka prepared interesting questions to warm up. And the answers and discussion was so interesting that we didn’t get beyond this opening slide. But that didn’t matter at all. Together in the group we shared our experiences on how to manage an upstream project and how to work with contributors. Tomas took notes and I’m looking forward to posting them somewhere.

The discussion was so interesting that I skipped “How to build Collaboration and influence Open Source Project” where I originally wanted to go.

On my lunch break, I walked around to four food trucks and chose a Japanese (or Vietnamese?) one where I had some great chicken pieces with rice Kaarage Don.

Then I watched the talk “Learning from Nix: how other package managers can do better”. Nix has a nice handling of hermetic builds, and Evan showed a mockup of how a similar thing could be done in DNF. Plus there are folks at Conflux working on similar things, so maybe we can look forward to a similar thing in the RPM world soon. Otherwise, the NIX packaging language looked very complicated. SPEC file is a trivial thing compared to that. In follow up discussion Daniel Riek mentioned that he has a quick'n'dirty tool that rebuild packages for specified AWS flavour - this gives him 10-20% performance boost. He promised me that he will clean up this script and next year he will present it at DevConf. :)

Then I looked at “Do you like your changelogs”. In the talk, Laura and Frantisek gave a live demonstration of how developers think about and work with changelogs, and combined it with data Laura had collected in her master’s thesis. The talk was entertaining and informative. We will definitely use the data from it in the Packit team, as automating Changelogs is something we want to pursue in the near future.

In the followup discussion about Towncrier I talked with Sviatoslav Sydorenko who pointed me to Chronographer GitHub CI application that checks if commit has towncrier entry. And to sphinxcontrib-towncrier that makes nice documentation [example]. We use towncrier in Mock, this will be useful.

I only hopped on the RPM developers meetup for a bit because I had to move and get ready for my own talk. It was called “Indiana Jones and obsolete projects”. The room was fully packed. In addition to the actual content, I had prepared a theatrical insert: I was dressed as Indiana, and I had a bullwhip. Tomas helped me with the dramatic entrance and a moment later he purposely gave me an excuse to try out the bullwhip. At the end we paraphrased the famous scene where Indiana pulls a gun against the sword. Content-wise, I went through several innovations: cell phones, smartphones, SSDs, CDs… and showed how one technology replaced another despite people initially rejecting the new technology. Similarly, I then thought about the life cycle of software projects. The biggest compliment for me was Martin’s comment, “This was the first lecture where I didn’t fall asleep today.” You can check the recording and the slides.

I stayed in the same room to listen to “Let the user speak”, where representatives of different projects shared their experiences with Packit, Testing Farm and tmt. Which for me as a member of the Packit team was pleasantly refreshing.

But after this session was over, I found myself terribly tired. So I staggered home and hit the bed.

I am a contributor to fedora-license-data upstream, and I investigated what will be the easiest way to build a new version of the Fedora package. I ended up using Packit for that. With just one command, it creates a pull request in src.fedoraproject.org.

At the start, we have to configure the upstream. I run:

$ packit init

in the upstream repository. This command created .packit.yaml with

# See the documentation for more information: 
# https://packit.dev/docs/configuration/ 

specfile_path: fedora-license-data.spec 

# add or remove files that should be synced 
files_to_sync: 
   - fedora-license-data.spec 
   - .packit.yaml 
# name in upstream package repository or registry (e.g. in PyPI) 
upstream_package_name: fedora-license-data 
# downstream (Fedora) RPM package name 
downstream_package_name: fedora-license-data

The upstream uses tags to mark a release. The commits look like this:

*  e960257 (tag: v1.0) Merge branch 'noarch' into 'master'

I put in .packit.yaml one line to configure how we do a release:

upstream_tag_template: "v{version}"

The packit.yaml is highly configurable and you can see result of my .packit.yaml.

Now I can run command:

# packit propose-downstream

If you did not git-push it, you have to run:

# packit propose-downstream --local-content

For some unknown reason, Packit still cannot figure out the version automatically. So I had to run:

# packit propose-downstream --local-content . 1.0 --dist

This command creates PR for all branches. To limit for just some branch(es) you can use the command line option --dist-git-branch rawhide,f36.

You can see the resulting PR here

By the way, the Packit team is working on a feature where you can configure this workflow in dist-git, and you will not need to modify the upstream - if you are not a direct upstream contributor.

How to set up httpd to survive when OOMKiller kills one of its children.

In Copr, we have had a leaking process in our frontend. It is one route, which was leaking few megabytes. The route has a separate child process in httpd, so only one process has been leaking. We still did not identify the culprit, and in the meantime we had to fight with OOMKiller.

Few megabytes here and there and the process was too big. And we run out of memory. OOMKiller came and killed the process (as it was the biggest one). Usually, you will not care. Httpd is killing its children periodically, and when one is killed, the master process starts new child immediately. But…

Default OOMPolicy is stop. And here I will quote from man systemd.service:

OOMPolicy= 
Configure the Out-Of-Memory (OOM) killer policy. On Linux, when memory
becomes scarce the kernel might decide to kill a running process in order to free up 
memory and reduce memory pressure. This setting takes one of continue, stop or
kill. If set to continue and a process of the service is killed by the kernel's
OOM killer this is logged but the service continues running. If set to stop the
event is logged but the service is terminated cleanly by the service manager.
If set to kill and one of the service's processes is killed by the OOM killer
the kernel is instructed to kill all remaining processes of the service, too.
Defaults to the setting DefaultOOMPolicy= in systemd-system.conf(5) is set to,
except for services where Delegate= is turned on, where it defaults to continue.

Stop means that the OOMKiller will kill the process, it master process and all siblings. Effectively systemctl stop httpd. :((

Do you want to try it? Grab mod_oom from Copr project.

Put in httpd.conf:

 
  LoadModule oom_module modules/mod_oom.so 
   
  SetHandler oom 
  

And visit http://localhost/oom. It will eat all your memory, and you will see what happens. :) Your httpd service will be stopped.

How to improve this?

You can put: OOMPolicy=continue in httpd.service. In fact, Joe Orton did it in Fedora as default. More information in BZ 1947475.

Next time you will have OOM in your child proces, you will likely not even notice. Unless you carefully check the log.

Big thank goes to Joe Orton and Pavel Raiskup for the investigation.

In previous blogpost, I wrote How to generate OpenPGP record for DNS (TYPE61). You may get puzzled what to do when you have different GPG keys with the same email. E.g. EPEL GPG keys are:

pub   rsa4096 2019-06-05 [SCE]
      94E279EB8D8F25B21810ADF121EA45AB2F86D6A1
uid           Fedora EPEL (8) 
pub   rsa4096 2013-12-16 [SCE]
      91E97D7C4A5E96F17F3E888F6A2FAEA2352C64E5
uid           Fedora EPEL (7) 
pub   rsa4096 2010-04-23 [SCE]
      8C3BE96AF2309184DA5C0DAE3B49DF2A0608B895
uid           EPEL (6) 

Three different GPG keys with the same email. How should we put it in DNS?

This is actually nothing unusual in DNS. You can have multiple DNS entries normally. E.g.:

;; ANSWER SECTION:
seznam.cz.        274    IN    A    77.75.74.172
seznam.cz.        274    IN    A    77.75.74.176
seznam.cz.        274    IN    A    77.75.75.172
seznam.cz.        274    IN    A    77.75.75.176 

When you run the command suggested in the previous blogpost, you will get:

$ gpg2  --export-options export-dane --export epel@fedoraproject.org
$ORIGIN _openpgpkey.fedoraproject.org.
; 94E279EB8D8F25B21810ADF121EA45AB2F86D6A1
; Fedora EPEL (8) 
1a355c3f6ac5389917041321fdddee2c0ffc4a38f78adec159a015ec TYPE61 \# 1141 (
        99020d045cf7cefb011000c93882169651ae7719e9bc99e4c50cf60ada1623b8
        287559e8725add97cde4563a92429fb6760c6e1b99948800d47d81da450cf12b
        0f1e7ee427c31cd4f6467bd27802c6d99b4161a65267d24e189aa4ecf4d34d7c
        f9ea3930569b776bdd886a35cbee759b6b110e937ca9d09aa97928eb973232e2
        d7ae88c91c8baf440ff1ee2a8ead17f26bcc773b10b83c4825e698039cff2954
        ad252d89dc0c440237be83f6e6e16505a121217fcc923e7bcd3a57bd61cdfc8b
        fccb779909bf962fa544a536e54b24f5d59a7f8347ff06473083c0915278b83e
        07fee4b8f70a969f28936064fb8546e279c17b72b84b0a6951cef251f269113f
        aff84ff177b4d0cf5997833440d5154913147354ebf876f8edfdf0c358fdf3a1
        c8a68d2b79e713483a409d5d387df59571c0465a453ad5addd599953426758b7
        9876f6a9e047dff6a4d6649848ec2ab45a6f0380b5295e0365926aaebc4ecfd9
        6e4402bd84e40a8a4280db7f0fc6896751bc758a78d18c44b9c623742ea17dd4
        a409570fbbf5dd5759c4dc9dbe97b2a5b7ff01f472ac744a864523ddc535a589
        0cdc335913bec2e4951eebfd27c5bb7811351905380b8182463fc73b3db9159c
        ee88fca80bd63c3eecc5ba033f0ec7e45764bc8631599b5bf4cc73c85a25f28b
        328528dd3564f273e2521a0e0b4fe423b8fd8a516b071054e5d52d1782130167
        3cc671ba8c9b3f22cbd1a50011010001b4284665646f7261204550454c202838
        29203c6570656c406665646f726170726f6a6563742e6f72673e890238041301
        02002205025cf7cefb021b0f060b090807030206150802090a0b041602030102
        1e01021780000a091021ea45ab2f86d6a166a00ffe319cb5b0b37e0607254342
        48e9ae4d1f5fb2328699bf53b06c2072aca472cbf98e3abc000663ff6f32f744
        d1f72bf44936669ef31354569920b3cf35204c6e811684c6d47e05868ffb67c0
        572bf8f26e8c174997ea5e74ce7e14856a5c0377095226f7d8355f4c6f2bdbc4
        df2a651535cfb3c4599ac8cc50c34e815193447bd731e99cf4cd209e7f6174f0
        36a53d3b1208f213108f8b9f175d74e34009419edbaf7af37ab97204e07e25dd
        7348f56b00fa5e332269a5614748300858aeb1f9a9b5dd52089d7b6a07f0b3e0
        60472c52b9776fc862ca55c38ed2a258fc8e2144b6702aebed40ab9587de2078
        6c4d9eeb3d340d217dddfaf03b258ded698f22873642c35126181f242fa2b064
        00c30abcc406c71017aec8c1bc1998f3c68860d07e1b39212e8d52d38c7d716c
        01b80e78563bfd555cb5fe2d710bc9c3bc6504dc8e3ff80102aea38c2f30faef
        f40d7af681131be57042535def85413c60af867cf019735b4cafca1f5f96447d
        1849bfb8d5f941dcd2a131253a746234486b48c4348785554d9b1dbfe97893f5
        4ff39eb66c1b92b0f2721b6c5cfef4c19451680932c775665f51f20cfb57d7e9
        9fb2fbcb92e127c1ce08cb18fd9effec752cfadfc0d4c23883b43983952e4997
        608b4cf1ba4896b8290b7808803f8cbbcca2b83a327ae383437160876b7eb39f
        0566933de2c4bf10d1ab91d793aaab606347f7f2a2
        )

$ORIGIN _openpgpkey.fedoraproject.org.
; 91E97D7C4A5E96F17F3E888F6A2FAEA2352C64E5
; Fedora EPEL (7) 
1a355c3f6ac5389917041321fdddee2c0ffc4a38f78adec159a015ec TYPE61 \# 1141 (
        99020d0452ae6884011000b5529857c0ca8201aacf507fd9b0e16c95a6de4d53
        b6a439396273bde9ffab81907bc40ac139279093b07dd22a9227ce7f73bd8e02
        7e0e5d8bd3eb781f09e5e926ce4cede99790fb0d4165928eef7d956f80a92366
        8d85a199194b44697438eee02308fbefa7485ef70c34597348f8f4d0ddb102a8
        cc6e39675769f669b004e60aba569a8fbc55d5c9fad56bfb9a9688035667fa87
        6b7845da627eaf2a4c7b07154df1a42cfe4fafdd196286438d9941f4da2e70cc
        8d3a00b266340327af9086d7ea2655b731af6a76293dc596e17d110cf729a9f1
        4d664eb8df123896c67e63612bf58bb94bff31d25cbeb66988a24684d30c1b75
        4bbbe3461366309eb2ba185a2460e73b90db17cac49a15e44f8487eb58c060c9
        9fa1df5a14609ee751470bee278e73b5856d2ae94ac3c410a5dd924d6adc4100
        c96915a69cca285ff3c04d38c2044c41f5d933dfc0ebbaf93b2241ccb6c22a96
        400e40c76c5f57774e8bfa044d970e3206d331712ddb8919b57073feab21cf79
        4f4e798f7e84cf41eb8f17694c638e1f146a30ae66f5f9456dac71b439d2ef0e
        fb5aacd6ec78c5ac3d15793c76be78e31aa7211c44297c3a453b36fc2d316181
        889dc913547e5418df958b3b32cd57ea55dde437260d75505c1e95234ba9f41f
        b8544673ecdcc243631e1e723ef9bda1d4750487ea27ab0a18f19bb4f357a3b1
        11311ca95b2473d338b4b70011010001b4284665646f7261204550454c202837
        29203c6570656c406665646f726170726f6a6563742e6f72673e890238041301
        020022050252ae6884021b0f060b090807030206150802090a0b041602030102
        1e01021780000a09106a2faea2352c64e5c7c60ffe2ca6aa6c4e3b4333baa9ca
        d28b1caaee66dbee5a2aade517ef4fdc30f4651414c678659197e27517838f6a
        8b10cb6b2591f1d746fece64c8e8b70b4b97ffa8a7cda632460f8187bd1baea2
        942b5e05d41aeb3e2dbd79a29f1be1ae305b577411b66bb3ad3e6c37cae2a6ab
        d129bfa07498fe84e4a49dbd2452a895ad19c03ac114bfc03cd7244347a79ade
        f48b26dc2632769cb418b440e70a387cf079b8b75484b1140ef87628afff7cc8
        e36b7a342fe48dcfc3746836729f9031094f5107710771379c160a32e6e9918f
        df4166b79b47534efb8801ae2bc87ddbc3e1f24d7cd0475f08521fd00218236f
        1e75d58c7405f621d60292a44487a1f05af69f976d7c9105ed2117f066d7ec56
        ec5a8dbd91732b56036a1cdc839df0015683ee6e041db4964991564091a1eb32
        ef00cadc13eb643878daa6248d5a59b650348e63598a9ce912ae08aebf17df6a
        b61153f6024ed7bfb246e2fa52fdbfbe0cd547544677054b2b09548c63c08d87
        e5a4243058c0004b18aff6e3f260f1dc12b4ab6d11b4c2321f011defbacd4e48
        ec77c591d9a5715f9904ec0cfef355138bcd5f7c477270140c49e3fc6a78e378
        bc23c553a3ac4f795643cc3b8a5e68858f79c26ba37ebea593cf6413325d393c
        dc9fb0c16d4aeedf7090306299dc4bed463b02ec50db2f4d7bab14be65503f1c
        69327e2bb8729815f155276ea97978067ed4b97a0f
        )

$ORIGIN _openpgpkey.fedoraproject.org.
; 8C3BE96AF2309184DA5C0DAE3B49DF2A0608B895
; EPEL (6) 
1a355c3f6ac5389917041321fdddee2c0ffc4a38f78adec159a015ec TYPE61 \# 1132 (
        99020d044bd22942011000cb1a7523db8655296ee588537f240e4282dce53672
        aceec060edf55b356af2884dd445b9f5257beb7701ed90ac98f7afe27d9d4b77
        7944da0385eb56c6676096c0935e7bbe92a7d67e8ac3fc4505db1b98f08ffe01
        33d1caee9864b6a15527c55b6368df4e371fc51bc633c601c1717c871d020d95
        11023bdfd71452409ee2028e7ca9c1e75edc4e02f42601b47dcfa43f87f0fe63
        f3e1a08269d4e57854d1c26c2b3d33b1d4541a600b9dcf0dc4d442ec1c81e63a
        a50828f5e4f578b352655ac9e4172d8af97551c0fa3fa0881a491e4f680d86a8
        a77513e78145c65d6ed0ce879f7be7d542a2529bb4eadaaf68a95678e89f7ae3
        0682448b51c92a6e5b977164edf858ceb0b30d813f63250026c7e25de5baab69
        5e8dc1bdf9e4f870051f71dc32ec1d6ae971bfbcd829709f36849f82ba447e8d
        84c78226fc8dd676dc0c13f19b9f076add5273800c4b54585ccc31f03648e621
        d3c094d27dfe2c518e2a7876066d5bc55c042acda0bf8b843ac87e3be72dd46e
        bc11f2f4fdec085ec567271a1e53163fa4622ed1e710c19c6d9f10a501e2d9d4
        5e535c32afd42082e194fa3a925d86abd7211cb1d4466aca6934867cf906b06e
        6fdae1da13d744c4b4f02b852a079706efa930d88ed7d5f76942ff68b080eea0
        6201bd5eb3b857c755dfd15f9bba0b15fbd36e0e66bf843f13a016a3f8248e6b
        7191cba56f202f8c5b58010011010001b4214550454c20283629203c6570656c
        406665646f726170726f6a6563742e6f72673e89023604130102002005024bd2
        2942021b0f060b090807030204150208030416020301021e01021780000a0910
        3b49df2a0608b8951fc60ffc0b18fbfda8edfd7b26fd365af07ca754128d6d1f
        129dae1373f9762b3b8c950d305944f4aebcbdb26a879222140e2a134f7c4813
        f6676c6ec81c7e6a07c66195727fa56e1796b12bdc82b5eecf480bb7c4c618a1
        8644e0282d7a6e52c6f51cdb4a10ec0f438cf5b90da73b3e612d1c83395d08d8
        bc1857c0631888c1294305114c454ec2fb5ce664ac083f59b4c7d8f5b786b7d2
        5ad71103b118a2723707cd1ddfd7dfc2ced29b229b4b93e6663a8e3ee4ef5761
        74b4c84a2219dc070a288d664f1317ed1e92a1cca561f1f7433bfdcfd72d4593
        70a29fc51b08ef4c58c14d476edf57308036510f963b0703edc4cf817bb9ba05
        a7438e128bf86328b80446e0a999ed8531b8b9bf67ca2ba9219ddc90f1ebde75
        5c0d419c2fa9500ff9e498d54878b60fc75b873ce4a559fba88c0620cd11fa2c
        bec8760e051c7040d2da3b156f1d4171483b236224500e4f68fc3f9fad55dabc
        c01d0f0d21fcc91a67e07749f5162ab9abb83a48e5bb13967f4b91692fff4947
        02661fb5fca0443a672f047611640e4997f9da90283119bda96903b3aaaa4e70
        72af9722eeccee6ec9b3176a2a4b1314c62570655eb6db0127d76bfc86612006
        895ac6cfd084ebaefa74c966f05facdd75c4d77419ad79a396873a8f03a58e77
        c09234c65e3881ece50b11279df7a3115ed7cb9345b901bf6977bb4a0d1e901c
        8eeb8075df0a8a519d9d9555
        )

I.e., three entries for 1a355c3f6ac5389917041321fdddee2c0ffc4a38f78adec159a015ec entry.

RFC 7929 explicitly does not mention how to handle this situation. The only relevant part is Appendix A. on page 18. If you read it you may get a bit different result:

$ gpg2  --export-options export-minimal,no-export-attributes --export 'epel@fedoraproject.org' | wc -c
3414
#^^^ this is the value at the end of the first line (number of octets)

$ gpg2  --export-options export-minimal,no-export-attributes --export 'epel@fedoraproject.org' \
    |hexdump -e '"\t" /1 "%.2x"' -e '/32 "\n"' 
# this will give you the value of the key

When you concat two previous results you will get:

; 8C3BE96AF2309184DA5C0DAE3B49DF2A0608B895
; EPEL (6) >
; 91E97D7C4A5E96F17F3E888F6A2FAEA2352C64E5
; Fedora EPEL (7) >
; 94E279EB8D8F25B21810ADF121EA45AB2F86D6A1
; Fedora EPEL (8) >
1a355c3f6ac5389917041321fdddee2c0ffc4a38f78adec159a015ec TYPE61 \# 3414 (
        99020d045cf7cefb011000c93882169651ae7719e9bc99e4c50cf60ada1623b8
        287559e8725add97cde4563a92429fb6760c6e1b99948800d47d81da450cf12b
        0f1e7ee427c31cd4f6467bd27802c6d99b4161a65267d24e189aa4ecf4d34d7c
        f9ea3930569b776bdd886a35cbee759b6b110e937ca9d09aa97928eb973232e2
        d7ae88c91c8baf440ff1ee2a8ead17f26bcc773b10b83c4825e698039cff2954
        ad252d89dc0c440237be83f6e6e16505a121217fcc923e7bcd3a57bd61cdfc8b
        fccb779909bf962fa544a536e54b24f5d59a7f8347ff06473083c0915278b83e
        07fee4b8f70a969f28936064fb8546e279c17b72b84b0a6951cef251f269113f
        aff84ff177b4d0cf5997833440d5154913147354ebf876f8edfdf0c358fdf3a1
        c8a68d2b79e713483a409d5d387df59571c0465a453ad5addd599953426758b7
        9876f6a9e047dff6a4d6649848ec2ab45a6f0380b5295e0365926aaebc4ecfd9
        6e4402bd84e40a8a4280db7f0fc6896751bc758a78d18c44b9c623742ea17dd4
        a409570fbbf5dd5759c4dc9dbe97b2a5b7ff01f472ac744a864523ddc535a589
        0cdc335913bec2e4951eebfd27c5bb7811351905380b8182463fc73b3db9159c
        ee88fca80bd63c3eecc5ba033f0ec7e45764bc8631599b5bf4cc73c85a25f28b
        328528dd3564f273e2521a0e0b4fe423b8fd8a516b071054e5d52d1782130167
        3cc671ba8c9b3f22cbd1a50011010001b4284665646f7261204550454c202838
        29203c6570656c406665646f726170726f6a6563742e6f72673e890238041301
        02002205025cf7cefb021b0f060b090807030206150802090a0b041602030102
        1e01021780000a091021ea45ab2f86d6a166a00ffe319cb5b0b37e0607254342
        48e9ae4d1f5fb2328699bf53b06c2072aca472cbf98e3abc000663ff6f32f744
        d1f72bf44936669ef31354569920b3cf35204c6e811684c6d47e05868ffb67c0
        572bf8f26e8c174997ea5e74ce7e14856a5c0377095226f7d8355f4c6f2bdbc4
        df2a651535cfb3c4599ac8cc50c34e815193447bd731e99cf4cd209e7f6174f0
        36a53d3b1208f213108f8b9f175d74e34009419edbaf7af37ab97204e07e25dd
        7348f56b00fa5e332269a5614748300858aeb1f9a9b5dd52089d7b6a07f0b3e0
        60472c52b9776fc862ca55c38ed2a258fc8e2144b6702aebed40ab9587de2078
        6c4d9eeb3d340d217dddfaf03b258ded698f22873642c35126181f242fa2b064
        00c30abcc406c71017aec8c1bc1998f3c68860d07e1b39212e8d52d38c7d716c
        01b80e78563bfd555cb5fe2d710bc9c3bc6504dc8e3ff80102aea38c2f30faef
        f40d7af681131be57042535def85413c60af867cf019735b4cafca1f5f96447d
        1849bfb8d5f941dcd2a131253a746234486b48c4348785554d9b1dbfe97893f5
        4ff39eb66c1b92b0f2721b6c5cfef4c19451680932c775665f51f20cfb57d7e9
        9fb2fbcb92e127c1ce08cb18fd9effec752cfadfc0d4c23883b43983952e4997
        608b4cf1ba4896b8290b7808803f8cbbcca2b83a327ae383437160876b7eb39f
        0566933de2c4bf10d1ab91d793aaab606347f7f2a299020d0452ae6884011000
        b5529857c0ca8201aacf507fd9b0e16c95a6de4d53b6a439396273bde9ffab81
        907bc40ac139279093b07dd22a9227ce7f73bd8e027e0e5d8bd3eb781f09e5e9
        26ce4cede99790fb0d4165928eef7d956f80a923668d85a199194b44697438ee
        e02308fbefa7485ef70c34597348f8f4d0ddb102a8cc6e39675769f669b004e6
        0aba569a8fbc55d5c9fad56bfb9a9688035667fa876b7845da627eaf2a4c7b07
        154df1a42cfe4fafdd196286438d9941f4da2e70cc8d3a00b266340327af9086
        d7ea2655b731af6a76293dc596e17d110cf729a9f14d664eb8df123896c67e63
        612bf58bb94bff31d25cbeb66988a24684d30c1b754bbbe3461366309eb2ba18
        5a2460e73b90db17cac49a15e44f8487eb58c060c99fa1df5a14609ee751470b
        ee278e73b5856d2ae94ac3c410a5dd924d6adc4100c96915a69cca285ff3c04d
        38c2044c41f5d933dfc0ebbaf93b2241ccb6c22a96400e40c76c5f57774e8bfa
        044d970e3206d331712ddb8919b57073feab21cf794f4e798f7e84cf41eb8f17
        694c638e1f146a30ae66f5f9456dac71b439d2ef0efb5aacd6ec78c5ac3d1579
        3c76be78e31aa7211c44297c3a453b36fc2d316181889dc913547e5418df958b
        3b32cd57ea55dde437260d75505c1e95234ba9f41fb8544673ecdcc243631e1e
        723ef9bda1d4750487ea27ab0a18f19bb4f357a3b111311ca95b2473d338b4b7
        0011010001b4284665646f7261204550454c20283729203c6570656c40666564
        6f726170726f6a6563742e6f72673e890238041301020022050252ae6884021b
        0f060b090807030206150802090a0b0416020301021e01021780000a09106a2f
        aea2352c64e5c7c60ffe2ca6aa6c4e3b4333baa9cad28b1caaee66dbee5a2aad
        e517ef4fdc30f4651414c678659197e27517838f6a8b10cb6b2591f1d746fece
        64c8e8b70b4b97ffa8a7cda632460f8187bd1baea2942b5e05d41aeb3e2dbd79
        a29f1be1ae305b577411b66bb3ad3e6c37cae2a6abd129bfa07498fe84e4a49d
        bd2452a895ad19c03ac114bfc03cd7244347a79adef48b26dc2632769cb418b4
        40e70a387cf079b8b75484b1140ef87628afff7cc8e36b7a342fe48dcfc37468
        36729f9031094f5107710771379c160a32e6e9918fdf4166b79b47534efb8801
        ae2bc87ddbc3e1f24d7cd0475f08521fd00218236f1e75d58c7405f621d60292
        a44487a1f05af69f976d7c9105ed2117f066d7ec56ec5a8dbd91732b56036a1c
        dc839df0015683ee6e041db4964991564091a1eb32ef00cadc13eb643878daa6
        248d5a59b650348e63598a9ce912ae08aebf17df6ab61153f6024ed7bfb246e2
        fa52fdbfbe0cd547544677054b2b09548c63c08d87e5a4243058c0004b18aff6
        e3f260f1dc12b4ab6d11b4c2321f011defbacd4e48ec77c591d9a5715f9904ec
        0cfef355138bcd5f7c477270140c49e3fc6a78e378bc23c553a3ac4f795643cc
        3b8a5e68858f79c26ba37ebea593cf6413325d393cdc9fb0c16d4aeedf709030
        6299dc4bed463b02ec50db2f4d7bab14be65503f1c69327e2bb8729815f15527
        6ea97978067ed4b97a0f99020d044bd22942011000cb1a7523db8655296ee588
        537f240e4282dce53672aceec060edf55b356af2884dd445b9f5257beb7701ed
        90ac98f7afe27d9d4b777944da0385eb56c6676096c0935e7bbe92a7d67e8ac3
        fc4505db1b98f08ffe0133d1caee9864b6a15527c55b6368df4e371fc51bc633
        c601c1717c871d020d9511023bdfd71452409ee2028e7ca9c1e75edc4e02f426
        01b47dcfa43f87f0fe63f3e1a08269d4e57854d1c26c2b3d33b1d4541a600b9d
        cf0dc4d442ec1c81e63aa50828f5e4f578b352655ac9e4172d8af97551c0fa3f
        a0881a491e4f680d86a8a77513e78145c65d6ed0ce879f7be7d542a2529bb4ea
        daaf68a95678e89f7ae30682448b51c92a6e5b977164edf858ceb0b30d813f63
        250026c7e25de5baab695e8dc1bdf9e4f870051f71dc32ec1d6ae971bfbcd829
        709f36849f82ba447e8d84c78226fc8dd676dc0c13f19b9f076add5273800c4b
        54585ccc31f03648e621d3c094d27dfe2c518e2a7876066d5bc55c042acda0bf
        8b843ac87e3be72dd46ebc11f2f4fdec085ec567271a1e53163fa4622ed1e710
        c19c6d9f10a501e2d9d45e535c32afd42082e194fa3a925d86abd7211cb1d446
        6aca6934867cf906b06e6fdae1da13d744c4b4f02b852a079706efa930d88ed7
        d5f76942ff68b080eea06201bd5eb3b857c755dfd15f9bba0b15fbd36e0e66bf
        843f13a016a3f8248e6b7191cba56f202f8c5b58010011010001b4214550454c
        20283629203c6570656c406665646f726170726f6a6563742e6f72673e890236
        04130102002005024bd22942021b0f060b090807030204150208030416020301
        021e01021780000a09103b49df2a0608b8951fc60ffc0b18fbfda8edfd7b26fd
        365af07ca754128d6d1f129dae1373f9762b3b8c950d305944f4aebcbdb26a87
        9222140e2a134f7c4813f6676c6ec81c7e6a07c66195727fa56e1796b12bdc82
        b5eecf480bb7c4c618a18644e0282d7a6e52c6f51cdb4a10ec0f438cf5b90da7
        3b3e612d1c83395d08d8bc1857c0631888c1294305114c454ec2fb5ce664ac08
        3f59b4c7d8f5b786b7d25ad71103b118a2723707cd1ddfd7dfc2ced29b229b4b
        93e6663a8e3ee4ef576174b4c84a2219dc070a288d664f1317ed1e92a1cca561
        f1f7433bfdcfd72d459370a29fc51b08ef4c58c14d476edf57308036510f963b
        0703edc4cf817bb9ba05a7438e128bf86328b80446e0a999ed8531b8b9bf67ca
        2ba9219ddc90f1ebde755c0d419c2fa9500ff9e498d54878b60fc75b873ce4a5
        59fba88c0620cd11fa2cbec8760e051c7040d2da3b156f1d4171483b23622450
        0e4f68fc3f9fad55dabcc01d0f0d21fcc91a67e07749f5162ab9abb83a48e5bb
        13967f4b91692fff494702661fb5fca0443a672f047611640e4997f9da902831
        19bda96903b3aaaa4e7072af9722eeccee6ec9b3176a2a4b1314c62570655eb6
        db0127d76bfc86612006895ac6cfd084ebaefa74c966f05facdd75c4d77419ad
        79a396873a8f03a58e77c09234c65e3881ece50b11279df7a3115ed7cb9345b9
        01bf6977bb4a0d1e901c8eeb8075df0a8a519d9d9555
        )

This is also a valid result. I contacted the author of RFC 7929 Paul Wouters to clarify this and he suggested that the first option is preferred. But when working on implementation, you should keep in mind that the second is an option as well.

Yesterday, I wrote how you could verify packages using GPG stored in DNS. You may wonder how you can store it in DNS?

Easy but least secure way

Take the armored GPG keys. E.g., RPM-GPG-KEY-fedora-35-primary and you can copy it into this service. You will need to add the email associated with the key as well. Use: fedora-35-primary@fedoraproject.org. Click Generate, and you have your DNS record ready.

Hey, how should I know that email address? That is easy. Run:

$ gpg2 RPM-GPG-KEY-fedora-35-primary
gpg: WARNING: no command supplied.  Trying to guess what you mean ...
pub   rsa4096 2021-02-04 [SCE]
      787EA6AE1147EEE56C40B30CDB4639719867C58F
uid           Fedora (35) 

Hmm, I had to use the command-line anyway. Can I somehow generate it from the command-line, without using 3rd party service? Sure:

From Command Line

First, import the key:

$  gpg2 --import RPM-GPG-KEY-fedora-35-primary 
gpg: key DB4639719867C58F: public key "Fedora (35) " imported
gpg: Total number processed: 1
gpg:               imported: 1

And then export it:

$ gpg2  --export-options export-dane --export fedora-35-primary@fedoraproject.org
$ORIGIN _openpgpkey.fedoraproject.org.
; 787EA6AE1147EEE56C40B30CDB4639719867C58F
; Fedora (35) 
e27f1efe21ae589b7796e61af3ac4a4c1c2428615daca70d8f1c9e96 TYPE61 \# 1172 (
        99020d04601c49ca011000cb7fc60791ecc9e9a765318c3b6861889496a5b7c2
        63db1fc1d8afa121f22ec46b69563ede2d180353bea3693e69543e6614c277a6
        47a7791fdc4e6aaa42437242c857da8417c04cd449bd4234c09f0868245fc436
        cd992b21c0ab174436c2b29b95ab9d854fa255a9c00ec9b5f1812ab1be40537e
        ddb54accdd061d5a0f51f9788f7d5f112818d3d37bd504e74c4ac637f4b6829f
        a229d4ebe9f08128fc8784558d6a98238f01a2b46c91f7b9f8380ae7a4b3e4d2
        c105937822f1b992fa38c5e838f2b0bcc41fc5b8355c3f2fb99d0ab63fdc347d
        b16aed319ac02ef472697a4bce3d1c65caab63c997eba1589e172a70660bd0c2
        d9e733cfe0484bbf2554eb634e76984513ec271bef891c1c54c57cc9259e41bf
        d1f5380116b9d381f7c63d24b5b7b7bca70f5e83ceb8a055b074afd34506600c
        8f7aecbef3b714fd812133b9374952a3e9e21983288c3b4c25d5818344a72e97
        092cf40fae964835a136f8b37ce666f3f4ec6a13c56e368da2b9592f8d85979e
        f3ad17a1585b7352a94be8df6688ecac4b59f0f6d467e62f17c469add580ae31
        db264d89a51280c53871b1002c0611b5d0bbb1d9668c0748362393dc31d27f72
        a8e8d3c71cae3057ba2c56ae2e62bbd317a7ca93fdf4b3366b1d2209fcfea64a
        8bc42e95fbbbeeeaa15bfc2dd9bb678ad811b2a8e1c4cba3614e196f8864b45f
        21294fd75fbe36aa12f92b0011010001b4314665646f72612028333529203c66
        65646f72612d33352d7072696d617279406665646f726170726f6a6563742e6f
        72673e89024e041301080038162104787ea6ae1147eee56c40b30cdb46397198
        67c58f0502601c49ca021b0f050b0908070206150a09080b020416020301021e
        01021780000a0910db4639719867c58f8d600ffb058a609724806581e18f22a1
        ffe7faccf7d5bdb1f6d04ab7908ece1413749cb5fe054d66baf4bea93b92dd62
        eb0779b71ae96da4a44424a2ed9b9103d6b1b35c0a13d72a7d608f0b00374c08
        23b28c08eca1653ced852e0befdb9e3ad3791d8b44e09fc8f0bbe96f7ce62395
        9b94be6ec403c3b0b62eb95d00ce0c0eea326754a2b46699e0a40b56df9311ad
        788ac6121828f3a3b3d8f15dc93a4e7482a5a0f3637f6e6d84cd2ed3f375840d
        bd65be3e0896fc6022a29e835d735d8c66ad849924909fbf37fe94d4babb1807
        8f1fc9d32959d8b89b1abe86000c8ef545ab0194b048cd047234fcc040a7c644
        83bb2aa65b056f3415ee10857c39edd83225357c937cb1dec3a6c171e4cf9776
        7327412f87ae34cf283de65ab94076711385d615f23664a56843cacdfff7d78e
        864d2b2e7cb48cdc246ae7ff894f0e73a5ef3248cb44c260e54f552b226e596b
        73c179e330de5644506a2b8b19a2630c7ed6d31f5fb9c9b456fc0def05b6b145
        6daded4ccd8dccde91264489a2cdf4bfc7709088475405f3af360dd025105f5d
        93681089849001a689d09240cc7bd191124b2e02b66106221cd572daddc9c857
        ecada9b9c1209639f84f788c8b053649beba4d94f6de7ec4fde7b4dfd81348ba
        0cd9ac2e55c4348a919811875546bd22109c2cb3910cba114daa3e9896e7d1bf
        84f21766b2a870adca2d2f7f8b32504d196d8cc0
        )

And voilà, this your DNS record which you can copy'n'paste to your zone. In this case, to _openpgpkey.fedoraproject.org..

Remember, if the email associated with the GPG key is foo@bar.com then the generated record has to go to _openpgpkey.bar.com. zone.

And do not forget - all this has a sense only if your domain is signed using DNSSEC. Otherwise, anyone can falsify these records.

When you want to be sure that RPM packages installed on your system were not altered, you should enable

gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch

in your repo file or in dnf.conf. The problem is how to get the file on your disk the secure way?

We have fedora-repos and distribution-gpg-keys. This will allow you to fetch new keys. But how to fetch the very first GPG key?

You can either manually download, verify, and rpm --import that key. Fedora has a dedicated page for its GPG keys. This process is manual and boring. Can we utilize something which we can trust? What about DNS signed using DNSSEC?

Can we put the GPG key in the DNS record and sign it using DNSSEC? We can trust this record and can safely import it.

This is what Martin Sehnoutka has done in his master thesis Automatic verification of software packages with help of DNS. He even provided patches for DNF, created some DNS records, but then … things stalled. I picked it up a few days ago.

Fedora now provides GPG keys for Fedora 27+ in DNS. Want to check it?

Enable DNSSEC

Default Fedora installation use systemd-resolved. In such case, add to the file /etc/systemd/resolved.conf following line:

[Resolve]
DNSSEC=allow-downgrade

and restart the service:

sudo systemctl restart systemd-resolved.service

From man resolved.conf: If set to “allow-downgrade” DNSSEC validation is attempted, but if the server does not support DNSSEC properly, DNSSEC mode is automatically disabled.

Manual check

This step is only for curious nerds. :)

Start with the email associated with GPG key and use tool email2domain and run:

$ email2domain fedora-27@fedoraproject.org
2d81eb3c5ebd20d163ff111a2dbcdc7e3336825d7d2331a3ef543aa8._openpgpkey.fedoraproject.org

This is the DNS record to query and is formatted using RFC 7929.

Now query this DNS record:

$ dig -t TYPE61 2d81eb3c5ebd20d163ff111a2dbcdc7e3336825d7d2331a3ef543aa8._openpgpkey.fedoraproject.org

; <<>> DiG 9.11.26-RedHat-9.11.26-2.fc33 <<>> -t TYPE61 2d81eb3c5ebd20d163ff111a2dbcdc7e3336825d7d2331a3ef543aa8._openpgpkey.fedoraproject.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13055
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;2d81eb3c5ebd20d163ff111a2dbcdc7e3336825d7d2331a3ef543aa8._openpgpkey.fedoraproject.org.        IN OPENPGPKEY

;; ANSWER SECTION:
2d81eb3c5ebd20d163ff111a2dbcdc7e3336825d7d2331a3ef543aa8._openpgpkey.fedoraproject.org. 86400 IN OPENPGPKEY mQINBFiskqMBEADTbsoAXpDPk+FtcwBEPZQVe0YQYdOqavfQQVD/RYAc HnJW/K1bZhQusBjUIec9SfGi3uBNNmbziAvpd/ycMKyWHuWQLmBgbImr qnPbBPMXmxeNGnZjA1hjVDp0pzj2+gZQhqYWSf6kQy9u9A1mSU63Kl/t fw7+hX7Tc3I8feGAFCHcFQgESxUib8Mw/OOGR3Am9fKdA+K1kJeQIiZv XMcNFx+3CfoavhFdicuoT2KbcSuzRm76duKNHlLaP6/IbZxNiDWh8SDV pFaFPlqR/R/+wibA6e9wMf6CZ4vfUY7NKYf4tYBs0EYdkn3j/KhJJxdb +M46Q/xwq9ovZo7XIhLrIUPuMw91X9cbvkU/a9kE1ffdpNmF1fDnUcEk uuEqOl+aMVsUBEbAQ86yrwpDfL4XT9vwnDIkggKZyvDTZ6q00XKg3Ger KuZtQBl/YcHDXuBlB1fzpGl8a8hq/+GeT2sVxjlYwPXjrsKd1NeP6ctQ sR6gHuP3W5ohP4rArtM6ONN8rlTiodDLVGHBpUzIRdgr7RCL1AqB9vrd Q2MVZMasTcUnUvKo0H4mps+x05jGao0b0Z0TJc6wr6ybHH38NVG06VX5 rJZlfZchGwkWzmYxai55/7ln1vJmk+kbdS7pK0jfmeVJ8A77XLCL36oJ FCiCrYjV+ZGgvB+z08Jzwc7sgwARAQABtCxGZWRvcmEgMjcgKDI3KSA8 ZmVkb3JhLTI3QGZlZG9yYXByb2plY3Qub3JnPokCOAQTAQIAIgUCWKyS owIbDwYLCQgHAwIGFQgCCQoLBBYCAwECHgECF4AACgkQ9V50MPUoLuTc ww/+LFPVEyVguMeU/QABEsE5FEN7kcDReZtdwq7p/aKC29mzCxeHggit YOGlrINkJ26Aq6p+oW6w7JxBWJnKoTBYJDFzNIbp/6GbG4oKcEnWQZfT nRLTr5aukVdWBFevzC0huraobKz3joYRIX826VUzS/A418zpnDVPtpo3 x86V9f292rqi2tn9Q5cQC5Ck3/cjQDEwkN9gHz4j/c1oa6zBOcHbKkaZ dWA2dIs6XOxCIHg78i6VQwMMs+vfm2vbV3ACCcOVnd3d6NxIQuDLEQwd tdB2zI8R74bjacosrcafK+F2DnkM7WrLSCiTKLJBMRDx+X2nOjT5pLut s4FC/XYRO23SMtPAMzQ852Z1lkjsaVDsjzNqCasUB0vDPHLQE8aj0zch NBSzuHoKpXNYTyJztekWL/QXkUsXu0x7N5WhBlZ+lni6LtZU51l7BJd1 n+ZKnQ4gmjQ1ffVLbgzb9Z1MNje0s61FdKmUJQGULYqh32W4GV+RLvtI 6AJV/EmFCUEfRJ3eA8tJiyKe512wiim/WDhvzFuuPBup2Z3TufiaJQOy sLlN5+HUQitTtcd7j8ZgpsIAIZtSWOMxbIAJWbzn8gjfIj4ZDeo0ZZXH 0VgDkXtpv1g8R2aRAz6ob5YYW5VnI2UEr53x1Z7lmUhv/TUZn26IeU16 jCJ80k7pJvQSF8Y=

;; Query time: 560 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Čt úno 11 17:07:46 CET 2021
;; MSG SIZE  rcvd: 1272

This is huge! Where to look? First, we look for TYPE61 record. In ANSWER SECTION there is the GPG key of Fedora 27. It really exists there! But can we trust it? Every documentation about DNSSEC says that we should look for ad flag. It stands for “Authenticated Data”. This is the part:

;; flags: qr rd ra;

But the flag is not there. That is because I use systemd-resolved (default in Fedora) and it is not capable of forwarding RRSIG records to the client! Systemd has four years old issue which recently got some love and the situation is changing every minute. Hopefully, it will be resolved very soon now.

When we do not use systemd-resolved, will the situation be better?

$ dig @192.168.1.1 -t TYPE61 2d81eb3c5ebd20d163ff111a2dbcdc7e3336825d7d2331a3ef543aa8._openpgpkey.fedoraproject.org
....
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
...

The 192.168.1.1 is a DNS server on my home router. Use a DNS server of your provider to test this step. What about the flag? Yes! The ad flag is there. We can trust the data as they have been signed and verified from the top-level domain down to fedoraproject domain.

Security note: The response goes from resolver unprotected (unless you use DNS over TLS). You should really use resolver in network you trust.

To temporarily workaround this in DNF, I submitted pull-request.

I said that the bits are already in DNF. Want to try it?

DNF

Before we start, make sure you do not use systemd-resolved or apply this patch.

Put in /etc/dnf/dnf.conf:

[main] 
gpgkey_dns_verification=1

This option is well documented in man dnf.conf. I copy it here:

gpgkey_dns_verification
       boolean
       Should the dnf attempt to automatically verify GPG verification keys using
       the DNS system. This option requires libunbound to be installed on 
       the client system. This system has two main features. The first one is
       to check if any of the already installed keys have been revoked. Automatic
       removal of the key is not yet available, so it is up to the user, to remove
       revoked keys from the system. The second feature is automatic verification
       of new keys when a repository is added to the system. In interactive mode,
       the result is written to the output as a suggestion to the user.
       In non-interactive mode (i.e. when -y is used), this system will
       automatically accept keys that are available in the DNS and are correctly
       signed using DNSSEC. It will also accept keys that do not exist in the DNS
       system and their NON-existence is cryptographically proven using DNSSEC.
       This is mainly to preserve backward compatibility. Default is False.

Now, edit /etc/yum.repos.d/fedora.repo. And comment out the line with gpgkey:

#gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch

Delete the current GPG from your rpmdb. I am on Fedora 33. Run:

$ rpm -qa --qf "%{summary} %{name}-%{version}-%{release}\n" |grep pubkey| grep fedora-33
Fedora (33) <fedora-33-primary@fedoraproject.org> public key gpg-pubkey-9570ff31-5e3006fb

The pseudo-package with Fedora 33 gpg key is gpg-pubkey-9570ff31-5e3006fb. I can remove it from system:

$ sudo rpm -e gpg-pubkey-9570ff31-5e3006fb

Now choose some simple leaf package as testing object. I am choosing fedora-upgrade:

# sudo rpm -e fedora-upgrade

Now I try to install it using DNF. And now we can install it again and now it should be verified using DNSSEC. Let’s pass “-v” to have more fun (and information):

# sudo dnf -v install fedora-upgrade

The output is looong. I am going to pick up only interesting lines:

DNSSEC extension: Testing already imported keys for their validity.
Running verification for key with id: rpmfusion-buildsys@lists.rpmfusion.org
DNSSEC extension: GPG Key rpmfusion-buildsys@lists.rpmfusion.org could not be tested
Running verification for key with id: @copr#copr@copr.fedorahosted.org
DNSSEC extension error (email=@copr#copr@copr.fedorahosted.org): Email address must contain exactly one '@' sign.

At the start, DNF checks whether some GPG has been revoked - this was not possible until now. And it informs us that DNS entry for rpmfusion-buildsys@lists.rpmfusion.org cannot be found. The second part is an error because group projects in Copr contain two @ characters. This is bug of Copr, but it caused no issue previously, so we did not care. We will do about it in the future. Nevertheless, all these lines are normally suppressed and are just informative.

Dependencies resolved.
=====================================================================================================================================================================================================================
 Package                                                 Architecture                                    Version                                              Repository                                        Size
=====================================================================================================================================================================================================================
Installing:
 fedora-upgrade                                          noarch                                          33.2-1.fc33                                          updates                                           23 k

Transaction Summary
=====================================================================================================================================================================================================================
Install  1 Package

Total download size: 23 k
Installed size: 38 k
Is this ok [y/N]: y
Downloading Packages:
fedora-upgrade-33.2-1.fc33.noarch.rpm                                                                                                                                                799  B/s |  23 kB     00:29    
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                783  B/s |  23 kB     00:29     
warning: /var/cache/dnf/updates-0e22a1f5a0a34771/packages/fedora-upgrade-33.2-1.fc33.noarch.rpm: Header V4 RSA/SHA256 Signature, key ID 9570ff31: NOKEY
Fedora 33 - x86_64 - Updates                                                                                                                                                         1.6 MB/s | 1.6 kB     00:00    
Running verification for key with id: fedora-33-primary@fedoraproject.org
DNSSEC extension: Key for user fedora-33-primary@fedoraproject.org is valid.
Importing GPG key 0x9570FF31:
 Userid     : "Fedora (33) "
 Fingerprint: 963A 2BEB 0200 9608 FE67 EA42 49FD 7749 9570 FF31
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-33-x86_64
Verified using DNS record with DNSSEC signature.
Is this ok [y/N]: 

DNF downloaded the package and found that it contains a new GPG key and ask me whether it can be imported.

The line:

Verified using DNS record with DNSSEC signature.

is not in the current code. I proposed it new pull request.

When you use dnf -y DNF normally imports GPG key without asking. When you use gpgkey_dns_verification and the DNS entry exists, it must match the file before the key is imported.

And that’s all.

Future work

I will play with this a bit more. And you can too. I hope that this will leads us to future where we will do not need to distribute GPG keys as separate files.

In the near future, I want to provide DNS records for all Copr projects.

A few days ago, we announced no-cost RHEL. Here is a quick guide on how to get this no-cost subscription:

Do you have Red Hat account? No? Then navigate to access.redhat.com. On this page, click on the account icon in the upper right corner:

You will get the following screen:

Click on the “Register” button and follow the process. It is one screen only and you will be done within a minute.

Now navigate to developers.redhat.com and log in using your Red Hat account.

From the top menu choose Linux -> Download RHEL. That will get you to Download Page. You can download the ISO image. Or you can Set Up AWS EC2 Instance. Or you can use any other way. Your no-cost subscription is already active. You can check it on access.redhat.com/management:

When you click on “Active Subscriptions” and then on the Subscription Name “Red Hat Developer Subscription” you will see this page which describes your subscription:

On your system with Red Hat Enterprise Linux, you will run

subscription-manager register --username=admin --password=secret

And now, your system can consume all the content from Red Hat. Including the latest security errata.

This year, we had Nest conference instead of traditional Flock, which has been canceled due to COVID. The conference happened purely remotely over the Hopin video conference. This was good and bad. The good is that we saved a lot on traveling and that it happened at all. It would be bad if it was canceled. The bad part was that I found it hard to focus on the conference. There are too many distractions at home. It was much harder to socialize. And a lot of people had issues either with microphone or internet upload. It was sometimes hard to follow. The conference was organized mostly for US folks, and therefore some sessions were very late in my timezone.

There were a lot of interesting pools. Some of them:

The Lenovo ThinkPad laptops that are going to be released soon with Fedora installed currently all have Intel processors. Would you be interested in purchasing one with an AMD processor?

• 69 % AMD
• 21 % Intel
• 10 % Neither of them.

Which desktop environment do you use?

• 68 % GNOME
• 12 % KDE
• 4 % i3
• 4 % XFCE

Here are my notes from the session. The notes are probably very raw:

Fed Brunch = fbrnch, a helpful cli tool for Fedora Packagers

Lives at https://github.com/juhp/fbrnch.

A new tool (not yet in Fedora, but lives in Copr) to automate even more things than fedpkg. E.g., fbrnch create-review my-new-package.spec, fbrnch bugs [package]

For me, the most interesting part was when I found about rpmbuild-order, which sorts rpm package spec files using the build order. It can even find the loops. This is already in Fedora.

Lenovo & Fedora Q&A

In case you do not know yet - Lenovo created a small team focused on supporting Linux on notebooks. What is important - they introduced themself to the community, and they are easy to reach using email or forum and are working on broken things.

There is going to be a Fedora portal on the Lenovo web site. The URL is going to be announced. When you register with your @fedoraproject.org then you will receive a discount. The discount will vary and will change from time to time. But it should be very close to what Lenovo employee has as a discount.

Lenovo is working on LVFS very hard. No details or dates disclosed.

ThinkLMI - utility to access to BIOS WMI. It is being prepared to upstream but does not meet the quality to land in the kernel as a driver yet.

Work with upstream: Lapmode sensor enablement - accepted; Palm sensor, Performance mode control - in progress.

CPE Year in Review and Leadership Q&A

Fedora - 117 Physical systems, 250 Virtual Systems

And more for CentOS (I did not have enough time to copy the numbers from slides).

Rawhide gating completed late in 2019.

Noggin - replaces FAS2, uses FreeIPA for the backend.

CentOS Stream.

DNF counting - better statistics based on countme

Fedora-messaging.

Toddlers - small programms listening to fedora-messaging.

MBBOX - Module Build in a Box

rpmautospec - getting rid of changelog in spec files.

Monitor-gating - end to end testing of entire packager workflow.

Interesting number from Data Center move (107 servers moved, …)

CPE Survey

My subjective feedback is that CPE is corporately organized, which has good and bad consequences.

MBBox - Module Building in a Box

Koji{-hub,-builder,ra} and MBS deployment in OpenShift.

For me, the new thing was that templates in OS are deprecated in favor of Operators (ebook).

Rpmautospec: goal, design and scope

rpkg-util approach adds too much complexity.

Emulating a human, as good as possible.

New macros: %autorel, %autochangelog.

fedpkg build triggers Koji builder plugin, which ensures that git tag is written in dist git and runs rpmautospec, which generates missing changelog and passes it to build.

Known shortcomings: among other - scratch builds from local SRPM don’t work yet.

Source code: https://pagure.io/fedora-infra/rpmautospec

Meet your FESCo

Neal explained why he uses so many IRC nicks.

Kevin said that we should work on faster releases, increased pace without sacrificing quality.

There was a discussion on how are FESCO members influenced. By RedHatter and others.

It lasts 45 minutes before the first question about modularity pop up. Neal explained that it is floating, but much better than Software Collections.

Lightning talks

It happened in Mozilla Hubs, which is a kind of VR. Interesting experience.