03:33:53 Experimental (bcrawl) branch on underhound.eu updated to: 0.23-a0-5261-gd9800d219b 05:33:58 03nlavsky02 07* 0.35-a0-689-gad7af12997: feat: make &H clear a couple more negative statuses 10(2 minutes ago, 1 file, 2+ 0-) 13https://github.com/crawl/crawl/commit/ad7af12997cd 06:50:51 New branch created: pull/5334 (2 commits) 13https://github.com/crawl/crawl/pull/5334 06:50:52 03Zhang Haocheng02 07https://github.com/crawl/crawl/pull/5334 * 0.35-a0-689-g6182f960d0: Fix a typo The tripwire is 2000 turns, 500 should be the correct value. 10(2 hours ago, 1 file, 1+ 1-) 13https://github.com/crawl/crawl/commit/6182f960d02e 06:50:52 03Zhang Haocheng02 07https://github.com/crawl/crawl/pull/5334 * 0.35-a0-690-gd652fa767f: Don't wait infinitely to regenerate HP/MP in &O 10(30 minutes ago, 1 file, 1+ 1-) 13https://github.com/crawl/crawl/commit/d652fa767f75 06:54:05 03Zhang Haocheng02 07https://github.com/crawl/crawl/pull/5334 * 0.35-a0-689-g0ebe34a3c6: Don't wait infinitely to regenerate HP/MP in &O 10(76 seconds ago, 2 files, 2+ 2-) 13https://github.com/crawl/crawl/commit/0ebe34a3c6a5 06:56:14 03Zhang Haocheng02 07https://github.com/crawl/crawl/pull/5334 * 0.35-a0-689-g3fb10c51f6: Don't wait infinitely to regenerate HP/MP in &O 10(3 minutes ago, 1 file, 1+ 1-) 13https://github.com/crawl/crawl/commit/3fb10c51f695 10:12:11 <04dracoomega> Hmm... dungeon_char_type.h has TAG_MAJOR_VERSION stuff to dummy out unused entries, but are these even serialized anywhere where their exact value would matter? 10:13:33 <04dracoomega> Glyph overrides for .rc files are done by name 10:18:14 <04dracoomega> Yes, people have inserted new enum members multiple times, even if they only dummy out rather than remove old ones. But if insertion doesn't break anything, removal shouldn't either. 10:18:34 <04dracoomega> They do just as much to change the actual enum values 10:19:36 <04dracoomega> (And I continue to be unable to find any place these could be serialized in any way. It's not impossible I'm overlooking something, but the combination of not finding it and seeing people having mucked with the enums in recent years makes me think this dummying-out was unnecessary.) 10:21:56 <04dracoomega> It would be nice, at least in rare circumstances (eg: a couple XL-related queries are actually somewhat important to my current mini-project) 10:22:33 <09gammafunk> ok, I do want to work on long term improvements as mentioned, but I can think about how a query system without timeouts would work 10:22:41 <09gammafunk> I'm actually wondering what kind of "UI" it should have 10:23:12 <09gammafunk> like just a per-user toggle? you run command to toggle no timeout 10:23:45 A command flag might be easier 10:24:11 <09gammafunk> might be, yes, as an arg to lm/lg 10:24:13 yeah 10:24:59 <09gammafunk> then it just checks a special permission set...hrm 10:26:11 <09gammafunk> there's a wrinkle there about irc vs discord, since most people don't even use irc 10:26:53 * perryprog cries 10:27:23 <09gammafunk> it might just need to be a flag for !RELAY and each bot manages the permissions of whether to send that flag on a per-user basis based on a toggle 10:28:14 <09gammafunk> if we'd all like to migrate to irc and abandon discord, things become simpler! 10:28:35 <09gammafunk> anyhow, I'll poke at that and see if I can whip something up, sounds useful even in addition to sequell improvement 10:32:49 <04dracoomega> It definitely could be helpful. (Standard timeouts are not so bad many times as they can also be a hint to try narrowing the query a bit more, but sometimes there is really no other good way to get the data in question.) 13:49:15 <04dracoomega> @Odds Incidentally, I was thinking that perhaps instead of manually finding and flagging places where monsters should be reset immediately instead of deferred, KILL_RESET and its like should just always reset immediately. I think possibly every case you've passed an argument to do an immediate reset is using that and this is basically what all the 'behind the scenes' removal of monsters uses. 13:56:30 <11Odds> I had that thought but kill reset seemed very common including stuff like dispelling summons that I wasn’t so sure meant we should reset immediately 13:57:04 <11Odds> So while all behind the scenes removal uses it, I don’t think all uses are behind the scenes 14:17:15 <04dracoomega> I guess that is true, yes 14:18:33 <04dracoomega> Seems more scary on further consideration 14:18:37 <04dracoomega> Even if it would be nice ^^; 14:52:54 New branch created: pull/5335 (1 commit) 13https://github.com/crawl/crawl/pull/5335 14:52:54 03Aliscans02 07https://github.com/crawl/crawl/pull/5335 * 0.35-a0-690-g560f03c6b2: Add a flag to friendly battlespheres if they can see a target. 10(34 minutes ago, 3 files, 31+ 9-) 13https://github.com/crawl/crawl/commit/560f03c6b2fa 15:11:46 <11Odds> Yeah it would certainly be nice if we had a clean category of "not really a death" deaths to use this on 🙂 15:16:44 <02Monkooky> @Odds if you manage to get toroidal vaults working, that might also do a lot to solve the problems with desolation 15:17:45 <11Odds> Heh. I did actually have a little go at that and it's surprisingly kinda working (but would need a looooot more effort to be actually working) 15:19:05 <11Odds> Welcome to infinite V:5 15:19:06 <11Odds> https://cdn.discordapp.com/attachments/747522859361894521/1524902743485972612/image.png?ex=6a516fd9&is=6a501e59&hm=e9f1f68d621107189d808e4bd693ae5e1a3ee491a4a5db206603dcf8972c71bc& 15:23:09 <02Monkooky> oh that's rad 15:23:24 <09hellmonk> New gamemode 15:26:49 <11Odds> (The basics of travelling, fighting, and shooting across the seams work, but I'm sure a lot of things still don't and I'm not intending to make them, this was just a fun experiment) 15:42:00 <07wizardike> !lm * s=type type=orb.destroy 15:42:02 <04Cerebot> 16 milestones for * (type=orb.destroy): 16x orb.destroy 15:42:11 <07wizardike> !lm * type=orb.destroy 15:43:06 Unstable branch on underhound.eu updated to: 0.35-a0-689-gad7af12997 (34) 15:45:12 <04Cerebot> 180s limit exceeded: killed !lm * type=orb.destroy 15:45:36 <07wizardike> !lm * s=type type=orb.destroy x=avg(xl) 15:45:38 <04Cerebot> 16 milestones for * (type=orb.destroy): 16x orb.destroy [24.25] 15:46:21 <07wizardike> !lm * type=orb.destroy x=avg(xl) 15:46:24 <04Cerebot> 16 milestones for * (type=orb.destroy): avg(xl)=24.25 15:52:34 <07wizardike> It's very strange that this times out as we don't seem to have any trouble finding all the milestones of this type in the other queries 15:58:23 <09gammafunk> oh interesting...what the hell is "type" 15:58:27 <09gammafunk> !kw type 15:58:28 <04Cerebot> Built-in: type => verb!= 15:58:34 <09gammafunk> ok 15:58:43 <04dracoomega> Wow, even if only half works, I'm quite impressed. (I had wondered the other day, when this was last brought up, if periodically 'recentering' the map around the player (and moving everything else) would be easier than making the coordinates themselves actually wrap for all purposes, but it still also sounded like a terrible mess) 15:58:53 <09gammafunk> I see, I'm guessing what's happening here is that there's no noun 15:59:04 <09gammafunk> you're querying not for the combination of a verb and noun, but just for the verb 15:59:26 <09gammafunk> hrm, but it depends on the form of the query, I see 15:59:34 <09gammafunk> a summary is fast 15:59:41 <09gammafunk> so... 16:00:12 <09gammafunk> !lm * recent s=verb br.end=orc x=avg(xl) 16:01:14 <09gammafunk> well one thing I'll also note here, is wizardike's example query only has 16 matching milestones 16:01:41 <09gammafunk> and the other thing is that the first query that times out is retrieving everything from the matching entry 16:02:22 <09gammafunk> those summary queries actually limit themselves to certain fields, and this does matter at least in part because each field generally results in a join to a value table 16:02:45 <09gammafunk> for most anything that's a string, what's stored is an ID that's a row in a value table that has to be joined to 16:03:13 <04Cerebot> 180s limit exceeded: killed !lm * recent s=verb br.end=orc x=avg(xl) 16:05:10 <09gammafunk> so if you do a summary query, it actually doesn't do nearly as many joins etc; that said, when I look at the query analyses, what takes the most time is just the index scan on the verb_milestone index, which is an index on the combination of verb and milestone values; specifically it's taking lots of time for the read operations. It's slow at reading through all the records on the drive (since the attached storage is ultimately just 16:05:10 not very fast due to how attached storage tends to be for these cloud computing setups) 16:05:47 <09gammafunk> yeah and adding a weird summary didn't make draco's original query fast there 16:07:40 <09gammafunk> so the slow read is a basic problem that could be mitigated with my ideas for upgrading postgres and configuring it to do parallel IO more aggressively (something postgres 18 has). my testing with the attached drives shows that one can get nice read speeds similar to those advertised either if you have a massive block size for your reads (something not possible with posgres as it's not appropriate for a db) or lots of parallel reads 16:07:41 (something that postgres 18 can help with potentially) 16:08:40 <09gammafunk> but as far as why some queries get a lot faster, I'd guess that they're probably inherently smaller queries that are just getting slowed down a lot by dealing with more IO from reading more fields and doing more joins etc 16:09:49 <09gammafunk> so if you go from general query that looks at everything: !lm * type=orb.destroy to one that summarizes and hence only looks at a field or two: !lm * s=type type=orb.destroy x=avg(xl) 16:10:02 <09gammafunk> you can speed up a small query enough that way where it can matter 16:10:29 <09gammafunk> but for e.g. DracoOmega's query, that one is probably just pulling too many rows to get sped up enough; her query was already doing a summary and was timing out 16:11:36 <09gammafunk> I do think there's lots of room to improve the way sequell approaches queries even given how general it is, and if someone really has the expertise with DBs and postgres in particular, I'd encourage them to look at sequell and suggest changes...but beware, there be dragons... 16:12:47 <11Odds> (My rough approach is to make everything in the game route through a "geometry" object when it wants to work with coordinates, and use a prescribed set of methods you can then swap out in torus mode. I think in principle this should work fairly nicely, but of course there are a billion places that work with coordinates) 16:12:59 <09gammafunk> to be honest, I'm not sure that sequell's approach of making each unique string value a distinct row in a "value table" that one has to join to is the best approach, but greensnark knows a lot more about databases than I do 16:14:25 <09gammafunk> so this is a totally different technique than what the abyss does? 16:14:45 <09gammafunk> I've seen/read the code for that and it's frightening and it's kind of amazing that it actually works 16:14:53 <04dracoomega> It's very cool, really. (Like, I was never convinced this was a good direction for V:5 to go in in general, but the idea of being able to do this sort of thing would be neat. I assumed the cost involved would just make it a complete non-starter.) 16:15:29 <09gammafunk> abyss just kinds of shifts the player plus close by vaults back to the center as the player gets close to a wall, iirc 16:15:34 <11Odds> Yeah completely different 16:15:42 <09gammafunk> and it clobbers the rest of the level because it's the abyss 16:16:25 <11Odds> This is very much an actual torus and if the square to the left of (1, x) is (60, x) or whatever 16:18:31 <11Odds> (And yeah to be clear I'm not saying this is a good thing to do for V:5, but if we ever did want this I think it would be merely "very expensive" rather than "obviously prohibitively expensive") 16:19:41 <09gammafunk> thematically v:5 seems like sort of the opposite place where'd you want this...I guess if you like the idea of "at the bottom, the vault of treasures is so vast that it's infinite" 16:19:54 <09gammafunk> but that creates many obvious gameplay issues 16:19:58 <11Odds> And once we had toruses working klein bottles would be basically free! 16:20:26 <09gammafunk> well one thinks of a so-called "cone" geometry, really 16:20:44 <02Darby> yeah at least thematically desolation was the main place I originally found the idea interesting (since hugging walls or corners is often both good strategy and makes no sense) 16:21:29 <09gammafunk> I guess what I'm not thinking about correctly is that the level isn't infinite in the sense of "content" 16:21:34 <09gammafunk> it's just infinite in how you can move through it in a single direction 16:21:53 <02Darby> also curious what would happen if you shift-moved 16:21:54 <09gammafunk> so you're not adding infinitely more unique content to a level 16:21:58 <11Odds> Right, it's exactly the same size as vaults 5 (my screenshot is not very helpful and also not what it would look like if we actually did it) 16:22:18 <02Darby> (through an infinite corridor, I mean. would it eventually stop on its own, or?) 16:22:21 <11Odds> A fine question.... 16:22:58 <11Odds> Right now I'm guessing the answer is "you walk forever" or "the game crashes" 😛 16:23:09 <02Darby> probably 16:25:09 <12ge0ff> %git 4de224ac0 16:25:10 <04Cerebot> RypoFalem {CrawlOdds} * 0.35-a0-523-g4de224ac04: feat: Add options to limit longwalking aka shift running (5 months ago, 6 files, 102+ 3-) https://github.com/crawl/crawl/commit/4de224ac0425 16:25:31 <12ge0ff> ^--- it should be limited to 15 tiles by default 16:26:19 <09gammafunk> an RC-induced game crash? prefect for when you want the option to reset a bad level 16:27:47 <11Odds> You can reset a level as long as you have an infinite, 3-wide corridor which will never have a monster in it. 16:28:08 <11Odds> This is also known as "having cleared vaults:5" 16:28:09 <02Darby> quite powerful 16:28:57 <09gammafunk> ah, but your forget my high stealth rating! 16:29:38 <09gammafunk> I guess travel doesn't care about the monster not noticing you, does it 16:29:50 <09gammafunk> ...although runrest ignoring monsters... 16:30:45 <09gammafunk> in the end, I guess the zot clock will ruin whatever fun you might have 16:36:38 <11Odds> @dracoomega have you thought about the monster reset change as much as you're planning to? If so I think I'll merge it tomorrow (with the honest expectation that a few problems then crop up, given the wide-ranging nature of it and some of the obscure things the tests revealed) 16:42:16 <04dracoomega> I haven't really taken a closer look at it than I had when we were talking about it yesterday. I can make a point to give it a little time again later tonight (though I suspect it's in that category where no obvious issues are going to pop out without stress-testing it) 16:43:05 <04dracoomega> (I spent the last couple days in a death battle with webtiles UI code, my great nemesis. Think I am past that now, though, at least ^^; ) 16:45:08 <11Odds> Thanks! 16:46:15 <11Odds> Yeah spotting problems with approach is what I'm keen for your look at 16:52:29 03Zhang Haocheng02 {CrawlOdds} 07* 0.35-a0-690-g744d22ced1: Don't wait infinitely to regenerate HP/MP in &O 10(10 hours ago, 1 file, 1+ 1-) 13https://github.com/crawl/crawl/commit/744d22ced11a 16:57:05 <07wizardike> !lm * type=orb.destroy noun="destroyed the Orb of Zot" 16:57:07 <04Cerebot> 16. [2018-01-24 09:15:13] TittyCrusher the Grand Master (L27 SETm of Cheibriados) destroyed the Orb of Zot (D:9) 17:05:55 (I've probably sent this before but it's worth sharing again. Hyperbolic geometry roguelike! https://www.roguetemple.com/z/hyper/) 17:07:14 <07wizardike> I don't think IO speed was the problem for my query. I think we just don't have an index that can find the first milestone of a given type. Finding the first milestone of a given type and noun is fast though. 17:27:23 <09gammafunk> IO speed is definitely the problem for your timed out query, but what's happening is that due to the formulation of the query, the IO plays out a lot differently. It seems that the backward index scan on time is a major culprit in this query, see this summary of the analyze-explain: https://explain.depesz.com/s/LDU4I The html tab has a breakdown of time spent by query step, reformatted query shows you the final formulated sequell 17:27:23 query, and stats gives you a nice summary of time spent 17:28:43 <09gammafunk> it's not that there's no index on verb, there is in fact a separate index on just verb by itself as well as the joint index on verb-noun as well as an index on noun by itself 17:29:04 <09gammafunk> but due to the general way in which sequell has to formulate queries, things get...complicated 17:29:43 <09gammafunk> in any case, if you see something obvious in terms of DB structure and query formulation, I'm certainly open to suggestions and would encourage you to look into the go-sequell repo and the sequell repo 17:33:07 <07wizardike> I'm looking though the sequell repo but I can't find where we define the indexes 17:33:16 <09gammafunk> it's in go-sequell 17:33:21 <09gammafunk> but here: 17:34:03 <09gammafunk> Indexes: "milestone_pk" PRIMARY KEY, btree (id) "ind_milestone_banisher_id" btree (banisher_id) "ind_milestone_br_id" btree (br_id) "ind_milestone_cbanisher_id" btree (cbanisher_id) "ind_milestone_charabbrev_id" btree (charabbrev_id) "ind_milestone_cls_id" btree (cls_id) "ind_milestone_crace_id" btree (crace_id) "ind_milestone_cv_id" btree (cv_id) "ind_milestone_explbr_id" btree (explbr_id) 17:34:04 "ind_milestone_fifteenskills_id" btree (fifteenskills_id) "ind_milestone_file_id" btree (file_id) "ind_milestone_game_key_id" btree (game_key_id) "ind_milestone_god_id" btree (god_id) "ind_milestone_hash_id" btree (hash_id) "ind_milestone_ltyp_id" btree (ltyp_id) "ind_milestone_maxskills_id" btree (maxskills_id) "ind_milestone_milestone_id" btree (milestone_id) "ind_milestone_noun_id" btree (noun_id) 17:34:04 "ind_milestone_nrune" btree (nrune) "ind_milestone_oplace_id" btree (oplace_id) "ind_milestone_place_id" btree (place_id) "ind_milestone_pname_id" btree (pname_id) "ind_milestone_race_id" btree (race_id) "ind_milestone_rstart" btree (rstart) "ind_milestone_rtime" btree (rtime) "ind_milestone_sk_id" btree (sk_id) "ind_milestone_src_id" btree (src_id) "ind_milestone_status_id" btree (status_id) 17:34:05 "ind_milestone_title_id" btree (title_id) "ind_milestone_tstart" btree (tstart) "ind_milestone_ttime" btree (ttime) "ind_milestone_turn" btree (turn) "ind_milestone_urune" btree (urune) "ind_milestone_v_id" btree (v_id) "ind_milestone_verb_id" btree (verb_id) "ind_milestone_verb_id_noun_id" btree (verb_id, noun_id) "ind_milestone_vlong_id" btree (vlong_id) 17:34:13 <09gammafunk> on the milestone table 17:34:31 <09gammafunk> there are more indexes, but as you see, verb as was as verb_id, noun_id are both defined 17:35:13 <09gammafunk> and verb id contains a unique integer value that indexes a table of verb values (ditto for the combination of verb, noun) 17:36:37 <07wizardike> Do we have any indexes that include verb_id and ttime? 17:36:53 <09gammafunk> nope 17:37:14 <09gammafunk> just index on ttime and I added a reverse index on ttime manually 17:37:25 <09gammafunk> because go-sequell does not do this by itself 17:37:44 <09gammafunk> which is "milestone_ttime_idx" btree (ttime DESC) 17:37:56 <09gammafunk> just just adding DESC 17:38:30 <09gammafunk> as for as combinations like verb, time go, it gets very tricky because combinatorally the number of indexes you start to need is very large 17:38:58 <09gammafunk> we have verb-noun because those are always together but it's not as clear for other random combinations 17:39:12 <09gammafunk> this gets into areas of db optimazation that I'm just not qualified to judge 17:40:26 <09gammafunk> table size is 33GB for context 17:41:12 <09gammafunk> and my disk is 85% full at 205GB (37GB remaining) so I don't have lots of room to play around with crazy indexes necesssarilly 17:42:09 <09gammafunk> however if a verb-ttime index would actually improve things, certainly possible to add that 17:46:04 <07wizardike> I'm not sure that it would. I still need to figure out what the fast verb-noun query used to make it fast 17:46:40 <09gammafunk> I guess what would help would be if I did analyze explain on the fast query 17:46:53 <09gammafunk> but of course the analyze-explain is set up to only occur on slow (enough) ones 17:48:27 <09gammafunk> I can do that at some point this weekend by temporarily turning it on for all queries and rerunning it 18:04:57 <06pleasingfungus> triumph! 18:09:43 <09gammafunk> ...which is why you complete a Zig before you go to Vaults:5... 18:12:15 <07wizardike> I don't seem to be able to find the index definition in here either 18:13:08 <09gammafunk> well the schema is assembled pro grammatically based on the yaml descriptions of fields, if that's not clear 18:13:17 <09gammafunk> but let me see if I can find the relevant go code at least 18:23:32 <09gammafunk> Yeah it's hard for me to give you one location in a source file that is responsible, but generally see https://github.com/crawl/go-sequell/blob/master/schema/schemasql.go#L154 where the index statement is assembled and https://github.com/crawl/go-sequell/blob/master/crawl/db/crawlschema.go for the process of turning the yaml representation e.g. here https://github.com/crawl/sequell/blob/master/config/crawl-data.yml#L924 into SQL 18:23:33 statements to define a schema 18:23:59 <09gammafunk> it seems to make a single index on every field with variation only by field type 18:25:48 <09gammafunk> I'm not actually sure what makes it create the verb-noun index, probably something in go-sequell itself. There's is this mysterious yaml entry but it doesn't seem to get referenced: https://github.com/crawl/sequell/blob/master/config/crawl-data.yml#L730 18:26:57 <09gammafunk> aha, it is referenced at https://github.com/crawl/go-sequell/blob/master/crawl/db/crawlschema.go#L160 18:27:15 <09gammafunk> but the table name is of course a variable, so that's how it knows to create that one joint index 18:27:58 <09gammafunk> greensnark was absolutely insane to have written all of this, but it's probably the type of thing he's used to making... 18:30:00 <09gammafunk> (and I still don't know any Go at all and I don't really know any ruby either, aside from what little I've gleaned from sequell and reading references when I have to, which hasn't made setting up and running newSequell ideal...) 18:32:05 <09gammafunk> I have half a dozen local commits to fix up the ruby source for modern Gems and to add some light functionality to the perl frontend in terms of permissions, but I haven't merged them to master yet. Once I fix up some issues to the perl side of things, I'll do that, but regardless an actual container would be really nice here. Sequell is just way too many moving parts 18:49:39 <07wizardike> !lm * crace="lava orc" x=avg(xl) 18:49:46 <04Cerebot> 69289 milestones for * (crace='lava orc'): avg(xl)=11.23 18:49:47 <07wizardike> !lm * crace="lava orc" 18:52:48 <04Cerebot> 180s limit exceeded: killed !lm * crace="lava orc" 18:53:01 <07wizardike> !lm * crace="troll" 18:53:27 <04Cerebot> 4297763. [2026-07-10 01:49:21] anbu the Ruffian (L4 TrFi) killed Terence on turn 1279. (D:3) 18:57:25 <07wizardike> We are scanning milestones in reverse order of ttime so if a milestone that satisfies the condition hasn't been generated in a while it is very slow. That makes sense to me, but what does is 18:57:55 <07wizardike> that this query is somehow fast 19:00:45 <09gammafunk> well also remember that we have both a forward and a reverse index on ttime. This is a special case for the ttime field because I manually added the DESC index for just that field 19:01:02 <07wizardike> !lm * cv=0.12 19:01:11 <04Cerebot> 173673. [2026-06-23 03:41:04] kraphead the Stinger (L6 SEVM of Okawaru) became a worshipper of Okawaru on turn 2604. (D:6) 19:01:30 <09gammafunk> !lm * cv=0.33 19:01:38 <04Cerebot> 3964114. [2026-07-10 01:48:10] doublebanjo the Lord of Darkness (L16 DrDe of Kikubaaqudgha) killed Mara on turn 30517. (Shoals:3) 19:03:16 <09gammafunk> I'm not sure about your query being fast, but DO's original question was about a particular kind of summary query: !lm * recent br.end=Orc x=avg(xl) 19:03:33 <09gammafunk> if you're aware of any conditions that get the same results and speed up this query, that would be helpful 19:04:11 <09gammafunk> this is querying for a comparatively recent set of milestones, but of course ttime isn't relevant here 19:04:37 <09gammafunk> I think in most contexts where we need help, it's either a summary query like this or a ratio query 19:05:07 <09gammafunk> vs queries that simply return one milestone row, those aren't as useful in practice 19:05:58 <09gammafunk> so I guess there's a COUNT(...) going on for ratio queries with two queries being done 19:06:21 <09gammafunk> and then summary queries are doing a postgres function instead of COUNT() (I think) 19:13:24 <07wizardike> That's a different problem. You could probably speed those up adding cv to the indices. I'm not sure whether this would would be worth while or not though and I would need to check what recent actually does 19:14:46 <09gammafunk> well add cv to which indices? You mean a joint index with cv and another field? cv already has its own index (see what I pasted above) 19:14:53 <09gammafunk> !kw recent 19:14:54 <04Cerebot> Keyword: recent => cv>=0.33 19:15:36 <09gammafunk> cv is a bit special in that it's this BIGINT thing that's used to that cvs can be ordered with alpha versions coming before non-alpha 19:26:03 <07wizardike> You would have to add it to a lot of joint indexes with other fields. That's why I'm not sure its worth while. DracoOmega's query was already not to badly optimized, it just queries a lot of entries 19:30:37 <09gammafunk> Yeah, from what I vaguely recall when researching this, a joint index will actually only help with all of the relevant fields are in the index 19:30:57 <09gammafunk> I think there's a problem where we often have arbitrary combinations of fields 19:32:18 <09gammafunk> like for DO's query we need something like (verb_id, noun_id, cv_id) 19:32:54 <09gammafunk> but I don't know if that's really accurate 19:36:13 <07wizardike> That probably is what we would need. I believe that index could also be used as a (verb_id, noun_id) index or even just a verb_id index with a small extra performance cost but couldn't be used is a (verb_id, cv_id) index 19:38:42 <09gammafunk> cv is definitely one of the more important fields where maybe we do want to consider a few more indices based on it 19:39:19 <09gammafunk> perhaps I should just add that 3-joint index for verb, noun, cv and see if it helps 19:42:40 <07wizardike> Maybe. I kind of want to do some more testing as querying the last milestone using verb and noun is strangely fast so I'm not sure if we are doing something special with this index 19:50:19 <09gammafunk> !lm * recent br.end=Orc x=avg(xl) 19:53:26 <04Cerebot> 149858 milestones for * (recent br.end=Orc): avg(xl)=14.11 19:54:52 <09gammafunk> @wizardike I added the 3-index and it manage to squeak out DO's query 19:58:43 <09gammafunk> @dracoomega if your queries of interest involve verb, noun, and version, you might try them again now. There's a new index for all three of those fields together, so maybe some of them will succeed (your example query just barely does now) 19:59:03 <04dracoomega> I'll take a further look, thanks! 20:00:18 <09gammafunk> oh, and I see something else that's concerning... 20:00:19 <04dracoomega> Wow, that was fast, actually 20:00:53 <09gammafunk> initially I was very confused as to the following query showing up in the explain-analyze log: Query Text: SELECT COUNT(*) AS fieldcount, l_milestone.milestone FROM milestone INNER JOIN l_milestone ON milestone.milestone_id = l_milestone.id GROUP BY l_milestone.milestone ORDER BY fieldcount 20:01:32 <09gammafunk> and then I realized that this is just the count result that's seen in the "149858 milestones..." portion of the title 20:01:56 <09gammafunk> this particular portion of the query was explain-analyzed but not the rest of the query, namely the thing that actually grabs the xl average... 20:02:27 <09gammafunk> specifically it spent 158 seconds on that count... 20:02:39 <09gammafunk> this portion of the query isn't even of interest here 20:02:56 <09gammafunk> hence removing it with a title change would presumably have sped up the query further 20:09:16 <09gammafunk> oh maybe that query is in fact needed for calculating the average 20:09:54 <09gammafunk> anyhow, maybe that triple index will help and hopefully it doesn't slow anything down 20:12:33 <04dracoomega> It definitely seems to make some of these wildly faster 20:12:45 <04dracoomega> Thanks a bunch 20:14:04 <09gammafunk> no problem, and thanks to WizardIke for the suggestion 20:24:07 <04dracoomega> @Odds So, I looked over all the code on your branch again. The one thing that did jump out at me after a little bit was that I think your use of MF_PENDING_RESET so that dead-but-not-reset monsters return true for invalid_monster() is possibly wrong? invalid_monster() is frequently used as a guard for 'this monster's properties are not safe to query', which seems like it should be untrue for these dead-but-not-reset monsters. And 20:24:07 there are cases in several places where we would avoid things like checking these monsters' attitudes, when it seems like we totally could now? (They still return false for alive(), but there are places already where we knowingly check stats of dead monsters - some in attack.cc, I think? - so it doesn't feel like it should introduce other problems.) Like, it's possibl some of these invalid_monster() checks should be checking whether they're alive 20:24:08 instead, but in other cases, I think pending-reset monsters should not count as invalid_monster(), if that makes sense? (Though feel free to point out something about this I've overlooked.) 20:25:59 <07wizardike> !lm * nrune=40 20:26:00 <04Cerebot> 1. [2008-02-01 22:38:11] Jovan the Farming Demigod of Death (L27 DgNe) found the Orb of Zot! (Zot:5) 20:27:38 <07wizardike> !lm * nrune=29 20:27:39 <04Cerebot> 3. [2010-05-16 15:31:46] 78291 the Farming Infernalist (L27 NaFE of Makhleb) found a demonic rune of Zot on turn 724849. (Pan) 20:28:37 <07wizardike> That seems to be another oddly fast query 20:28:58 <09gammafunk> !lm * urune=3 20:29:11 <04Cerebot> 2472908. [2026-07-10 03:27:32] wiznift the Slayer (L24 DsMo of Beogh) reached level 4 of the Depths on turn 59462. (Depths:4) 20:29:41 <09gammafunk> !lm * nrune=3 20:29:45 <04Cerebot> 2473905. [2026-07-10 03:27:32] wiznift the Slayer (L24 DsMo of Beogh) reached level 4 of the Depths on turn 59462. (Depths:4) 20:29:56 <09gammafunk> oddly fast compared to what? 20:30:25 <09gammafunk> there's a reverse index on ttime remember 20:31:23 <07wizardike> It returned an old game. So it can't be scanning though all the games in reverse time order to find the newest 20:31:57 <07wizardike> !lg * god=Wudzu 20:32:46 <09gammafunk> it returned an old game but there's condition on an indexed field (nrune) 20:33:50 <07wizardike> Right, but if it was scanning backwards in time it would be very slow like that god=Wudzu query 20:34:54 <04Cerebot> 558. estick the Carver (L11 OpGl of Wudzu), mangled by a yak (kmap: hangedman_ranch) on D:10 on 2016-11-20 22:57:01, with 8439 points after 11744 turns and 0:24:12. 20:34:56 <07wizardike> god is also in index field 20:35:10 <07wizardike> !lg * god=Wudzu x=avg(xl) 20:35:13 <04Cerebot> 558 games for * (god=Wudzu): avg(xl)=12.46 20:36:09 <09gammafunk> so now you're querying different tables 20:36:14 <09gammafunk> for logrecord we have 20:36:25 <09gammafunk> "logrecord_sc_idx" btree (sc DESC) "logrecord_tend_idx" btree (tend DESC) 20:36:25 <07wizardike> !lm * god=Wudzu x=avg(xl) 20:36:26 <04Cerebot> 8897 milestones for * (god=Wudzu): avg(xl)=14.7 20:36:37 <09gammafunk> I'm talking about your slow query 20:36:45 <07wizardike> That was meant to be lm 20:36:51 <09gammafunk> !lm * god=wudzu 20:38:53 <09gammafunk> yeah I'm not quite sure why this is fast for nrune/urune regardless of value....wait 20:39:09 <09gammafunk> the problem with wudzu is unique 20:39:26 <09gammafunk> for milestones, because we should have no wudzu milestones 20:39:51 <09gammafunk> where was wudzu, and experimental, right? 20:39:52 <04Cerebot> 180s limit exceeded: killed !lm * god=wudzu 20:39:54 <07wizardike> urune returned a recent value so it was always going to be fast 20:41:05 <07wizardike> We have 8897 milestones for god=Wudzu according to my earlier query 20:42:04 <09gammafunk> yeah, I see for wudzu but that is very odd 20:42:10 !lm * god=wudzu s=explbr 20:42:12 8897 milestones for * (god=wudzu): 8897x thorn god 20:42:17 <09gammafunk> and regarding "always going to be fast", I don't see how that's the case 20:42:47 <09gammafunk> we just saw recent queries being slow (before the 3-index was created, at least) 20:43:57 <07wizardike> Those were aggregate queries were the condition wasn't just one thing we have an index for 20:44:58 <09gammafunk> yeah but for milestones we have a forward and a reverse index for ttime, the field that's used for these sorts 20:46:01 <07wizardike> Forward iteration wouldn't helps us find the newest milestone though 20:47:03 <07wizardike> unless you iterated all the values to make sure there isn't a newer one, but that would be slow anyway 20:47:46 <09gammafunk> !lg * god=trog 20:47:55 <04Cerebot> 1726447. Kaeinara the Chopper (L1 HuBe of Trog), blasted by a dart slug (slug dart) on D:1 on 2026-07-10 03:45:25, with 2 points after 62 turns and 0:00:49. 20:48:07 <09gammafunk> !lg * god=trog 1 20:48:17 <04Cerebot> 1/1726447. duke the Ruffian (L6 TrBe of Trog), shot by a centaur (arrow) on D:8 on 2006-12-03 17:34:10, with 470 points after 1628 turns and 0:24:37. 20:48:45 <09gammafunk> !lg * god=wudzu 20:51:00 <04Cerebot> 558. estick the Carver (L11 OpGl of Wudzu), mangled by a yak (kmap: hangedman_ranch) on D:10 on 2016-11-20 22:57:01, with 8439 points after 11744 turns and 0:24:12. 20:52:39 <09gammafunk> so in the case of my first query, it's using the reverse index on tend, and in the second it would be using the forward index, right? 20:53:55 <09gammafunk> but for the third query, it's stuck using the reverse index but for an entry that's late into the index 20:55:08 <07wizardike> Right that seems correct to me 20:58:11 <09gammafunk> so I guess I don't know why your e.g. nrune=49 query is fast despite having a similar index issue to the god=wudzu query other than pointing out that there is no join necessary on id for nrune 20:58:23 <09gammafunk> it's just a straight integer value field with no other table involved for the conditioning 21:06:20 <09gammafunk> yeah the analyze explain on that !lg * god=wudzu query points to the culpret as being Index Scan Backwards on ind_logrecord_tend so it's definitely using that index 21:06:33 <09gammafunk> but it seems that it's still just slow 21:07:26 <09gammafunk> https://cdn.discordapp.com/attachments/747522859361894521/1524990403764424826/image.png?ex=6a51c17d&is=6a506ffd&hm=1b69a74863b2d73bbf093682a15d4ea0f0d2f9748427b6256f65ea1b54e6fa55& 22:36:05 Unstable branch on crawl.develz.org updated to: 0.35-a0-690-g744d22ced1 (34) 23:00:02 Windows builds of master branch on crawl.develz.org updated to: 0.35-a0-690-g744d22ced1 23:12:31 Unstable branch on cbro.berotato.org updated to: 0.35-a0-690-g744d22ced1 (34) 23:56:03 Monster database of master branch on crawl.develz.org updated to: 0.35-a0-690-g744d22ced1