[HN Gopher] Gemini last models: temperature, top_p, and top_k ar...
___________________________________________________________________
Gemini last models: temperature, top_p, and top_k are deprecated
and ignored
Author : greatgib
Score : 125 points
Date : 2026-07-21 21:27 UTC (20 hours ago)
HTML web link (ai.google.dev)
TEXT w3m dump (ai.google.dev)
| greatgib wrote:
| 1. Sampling parameter deprecation (temperature, top_p, top_k)
|
| temperature, top_p, and top_k are deprecated and ignored. In
| future model generations, supplying these parameters returns an
| HTTP 400 error. Remove these parameters from all requests.
| tolugenius wrote:
| > To improve determinism, define a system instruction with
| explicit rules for your specific use case.
|
| Is this guaranteed to work any better than top_k or top_p? This
| just sounds like making a smaller version of a Agent.md doc.
| janalsncm wrote:
| It is guaranteed to work worse than top_k=1, that's for sure.
| furyofantares wrote:
| Depends on what you mean by determinism.
| tough wrote:
| fwiw sonnet-5 also drops temperature (sonne-4 had it)
| aesthesia wrote:
| My guess is that RL training being done with particular
| generation parameters makes models much more brittle to changes
| in these parameters, and that's why we're seeing changes like
| this across model providers. But I don't really know.
| pixelmelt wrote:
| I'm inclined to agree given how unstable Gemma 4 is when not
| using the "official" sampler settings
| impulser_ wrote:
| Good. These have been basically useless for the past few
| generations of models, and most of the time made the model
| perform worst.
| kouteiheika wrote:
| Obligatory "The Conspiracy Against High Temperature Sampling":
|
| https://gist.github.com/Hellisotherpeople/71ba712f9f899adcb0...
| NooneAtAll3 wrote:
| where can one learn what top_k and top_p mean?
| cratermoon wrote:
| https://www.ibm.com/docs/en/watsonx/saas?topic=prompts-
| model...
| krapht wrote:
| ironically, any frontier LLM will easily generate a tutorial
| at any detail you like explaining what these are.
|
| if don't have time for that, just know that these are
| technical parameters that affect how likely it is an llm will
| produce the same result after being asked the same question.
| barkingcat wrote:
| Is an llm able to explain to itself what these parameters
| do, and change its own parameter settings?
| chronogram wrote:
| An LLM is stateless so every time you send a message a
| command is run with the parameters, the prompts, and a
| copy of the chat history when using one of those chat
| interfaces, and then it returns text. If the output
| includes explaining it to an LLM that would then be
| included in the copy of the chat history placed in the
| new command.
|
| It would be a bit like opening a text editor and typing
| in that you want to increase the font size. Someone
| external to the text would have to come along and click
| the font options.
| imperio59 wrote:
| https://www.longformexplainers.com/llm-inference/10-sampling
| nodja wrote:
| The posted answers are either behind a paywall or very obtuse
| so I'll just explain. I'll assume you know what tokens are.
|
| A models output is not a single token, but a list with the
| probability for all the tokens that it knows, so we need to
| use a sampler to select the token that it's going to be the
| next token in the sentence. For example a simple greedy
| sampler will choose the token with the highest probability,
| but samplers normally pick a random token weighted by
| probability. A model usually knows about ~250 thousand tokens
| and the probability of some of these tokens are gonna be
| high, but the vast majority is close to but not actually 0%
| so there's a chance the sampler might pick some random token
| that doesn't make much sense, so we filter tokens.
|
| top_k filters the tokens so that only the k top tokens are
| selected. So top_k=50 will filter those 250k tokens to only
| 50. This is assuming the list of tokens is sorted by
| probability.
|
| top_p filters the top tokens until a percentage is
| accumulated. So if for example if you set the top_p to 0.6
| and the model gave the top token a 0.5 (50%) probability and
| the second top token a 0.2, those 2 token accumulated to 0.7
| which is greater than what you set it to (0.6) so no more
| tokens are selected. If this ran after top_k=50 it'll turn
| the list of 50 tokens into one of 2.
|
| After each filter parameter is processed, the probability of
| the tokens is adjusted to sum to 1 (100%), Also note that
| order of operation here matters, i.e. top_p could be applied
| before top_k, but most providers follow what's on
| huggingface, I think I've only seen different implementation
| in certain local model hosting frameworks.
| NooneAtAll3 wrote:
| Thank you. So they are essentially a protection against
| spurious errors, cool
|
| I don't quite understand the point about order of
| operations - does it do normalization after every such
| filter pass? why not leave it to the end?
|
| also: what is top_a? I saw it being mentioned in the GP
| link
| nodja wrote:
| Yes, the whole list will always sum to 1 (100%) because
| there's lots of more sampling parameters. top_p, top_k
| and temperature are just the ones that affect output the
| most. Most parameters do math around assuming the list
| sums to 1 and order is not always the same, some software
| even lets you change the order around.
|
| top_a is not very common and is better explained if I
| explain how the much more common min_p works. min_p
| filters out tokens below a certain threshold. The formula
| is <filter threshold> = <min_p> * <top token
| probability>. So if the top token has 0.5 probability,
| min_p = 0.1 would cut out tokens below 0.05. This is a
| tunable that lets you filter out other tokens depending
| on how confident the model is.
|
| top_a is almost the same formula but you just square the
| <top token probability>. So <filter threshold> = <top_a>
| * <top token probability> ^ 2. This makes the filtering
| ramp up faster (cut out more tokens) if the model has a
| much more confident top choice, but keep more choices if
| the model is not so confident.
| kouteiheika wrote:
| > So they are essentially a protection against spurious
| errors
|
| Only coincidentally. Sampling nonsense tokens will
| certainly degrade its performance and/or brick it, but
| it's also there to encourage diversity.
|
| For example, imagine we have the following sentence:
|
| > The color of this ball is ____
|
| Now, what should the model predict for "____"? There
| isn't really a "correct" answer here. It can be "red", it
| can be "blue", it can be "green", or any other color. But
| it's definitely not going to be "bythaaia". LLMs output a
| probability distribution for the next token, so imagine
| this is the probability distribution that it outputs:
| red -> 60% green -> 19% blue -> 19%
| bythaaia -> 2%
|
| So how do we decide which token to pick? Simplest way is
| to always pick the most probable one (in this case:
| "red"). In this case we'd ideally want it to be able to
| output "red", "green" or "blue" (since all of those are
| reasonable), but never "bythaaia" (whose 2% is most
| certainly noise). So a sampler is essentially an
| algorithm which lets the inference engine pick the exact
| token to output from this list.
|
| > I don't quite understand the point about order of
| operations - does it do normalization after every such
| filter pass? why not leave it to the end?
|
| Because you can technically compose _multiple_ samplers
| at the same time in a pipeline, and in some cases their
| order can matter and give you a different result (or take
| less /more time to execute). To give you a generic
| example: imagine you have a list with numbers in random
| order. You can execute one of two operations on it: (1)
| sort it, (2) take the leading 10 numbers. If you first
| sort it and then take 10 leading numbers you'll get a
| different result than if you'd first took 10 leading
| numbers and then sorted them.
| vintermann wrote:
| I used to try to train models ages ago (that is, a couple
| of years), and I never found out how they got good results
| from top_k in the first place. My problem was that as soon
| as the model had generated an unlikely token, it was
| quickly steering towards spaces it had never seen during
| its training. It found itself in unmapped territory, where
| anything might happen. With high temperatures, it would
| then generate noisy garbage, with low temperature it would
| generate repetitive garbage. We tried to fight this with
| rollout strategies and backtracking, i.e. if you find
| yourself in a space where all tokens seem equally likely,
| you're probably in junk land.
|
| Our tricks didn't work very well. But I didn't manage to
| keep up well enough to learn _what_ worked.
| est wrote:
| wow I wish I knew this post before I installed various UIs with
| tons of temp/top-X sliders.
|
| They are extremely confusing.
| gdiamos wrote:
| thank god, these parameters are so confusing
| salamo wrote:
| Possible reasons:
|
| - They might be dynamically adjusting these at inference time
| [1]. For example, start with a low temperature and generate
| samples with increasingly high temperatures until one of them
| passes some quality gate.
|
| - They don't want you to fine-tune on high temperature
| completions (rejection fine-tuning). You could call this
| "rejection fine-tuning rejection".
|
| [1] https://rlhfbook.com/c/09-rejection-sampling#related-best-
| of...
| regularfry wrote:
| Couple of other more businessy reasons:
|
| - SynthID hides the watermark in the sampling RNG. No
| randomness -> no watermark.
|
| - If you want to distil on the model outputs, you want temp=0
| outputs. No temp=0 -> worse distillation.
| orbital-decay wrote:
| You probably don't want temp 0, especially with Gemini which
| often fails in the greedy sampling mode in practice, often in
| ridiculous ways (e.g. multiple thousand token loops). As
| another comment says it's pretty brittle and this is even
| reflected in their docs somewhere IIRC. You want the "normal"
| temperature (whatever it is) and oversampling if necessary.
|
| Certain open models have/had the temperature locked on the
| official APIs, I assume they just have sampling incompatible
| with static temperature or do some fancy speculative
| decoding. It's clearly not to hide anything, as the weights
| are open and there are always alternative providers.
| charcircuit wrote:
| Along with everything else. These parameters can make speculative
| decoding less accurate increasing the inference cost.
| nichohel wrote:
| "Last" or "latest"? Those are rather different.
| froh wrote:
| hehe... fwiw, the un-editorialized title is "Using the *latest"
| Gemini models"
|
| I speculate OP wanted to put focus on their chosen detail in
| the title...
| david-gpu wrote:
| The OP does not appear to be a native English speaker. And, for
| example, languages like Spanish don't clearly differentiate
| between "last" and "latest".
| henriquecm8 wrote:
| I am not a Spanish speaker, but Portuguese is usually
| similar. In it we say "ultimo" for last, and "mais recente"
| for latest, so while there isn't a single word translating,
| we still have its equivalent. I imagine it's the same for
| Spanish.
| herpderperator wrote:
| I'm curious: If someone wanted to serve models off
| hardware/silicon directly (like Cerebras or Taalas, and soon
| Google I think) rather than GPUs, would these parameters still be
| adjustable at request time? Or would they have to decide that
| before the model can even start serving and it would be locked in
| until they reload it (which would make it briefly unavailable)?
| StrauXX wrote:
| I'd expect that sampling would happen in software. Probably the
| hardware system would output a vector of probabilities over the
| tokenspace, just as the nets do when run in software.
|
| I don't know though and am not aware of any docs going into
| detail here. That being said, sampling is really cheap. So
| implementing it in hardware wouldn't be worth it.
| franky47 wrote:
| > To improve determinism, define a system instruction with
| explicit rules for your specific use case.
|
| "Please be deterministic".
| rf15 wrote:
| please oh god please, I don't know how to
| Zambyte wrote:
| In all seriousness, this seems like it could be fine if done
| well. You can just have a model do a pass over the system
| prompt and set reasonable parameters based on that. That's
| probably not what they're doing, but it could be.
| dev_dan_2 wrote:
| > improve determinism
|
| Oh well. I might be to picky here, but how I see things,
| determinism cannot be improved or worsened, but achieved or not
| achieved. Or Partially archieved, when analyzing a system that
| has both components that are deterministic or non
| deterministic.
|
| There are times to think in absolutes, and when talking about
| deterministic behavior of technical systems, this is one of
| them. Join the sith side, we have cookies, and when we _say_ we
| have cookies, we _do_.
| gregdaniels421 wrote:
| Probably what is meant is consistency instead of determinism
| bob1029 wrote:
| > To improve determinism, define a system instruction with
| explicit rules for your specific use case.
|
| What if I want to do the other thing? When performing research
| with many sub agents, having a lot of diversity in the hypotheses
| is a big deal. If my 5 parallel sub agents all produce the same
| conclusion I might as well have only ran one.
|
| The latest OAI models have done the same thing. I'm currently
| adding random variation to prompts to compensate for the lack of
| higher temperature sampling.
| orbital-decay wrote:
| Current post-trained models are stochastically and semantically
| collapsed to a large extent, you'll never get meaningful
| semantic diversity if your prompt stays the same. This makes
| the temperature brittle, the model becomes dumb much earlier
| than meaningfully diverse. There are fancy samplers that claim
| to overcome this (comments ITT mentioned them), but a) you need
| a local inference stack to use them and b) I've yet to see any
| definitive evidence they add any deep semantic diversity
| without dumbing the model down, most of what I've seen is
| pretty superficial, including distribution-aware samplers.
|
| >I'm currently adding random variation to prompts
|
| This is better than naive temperature tweaks, if done right.
| marcinjachymiak wrote:
| It's not surprising since these are all reasoning models now.
| Token sampling/decoding isn't exposed because the reasoning
| models are trained around specific configurations. That's why
| it's been replaced with "reasoning effort" instead (low, medium,
| high "thinking" levels).
| kittikitti wrote:
| The API endpoints are less useful if you can't even fine-tune
| these parameters. These endpoints used to be developer oriented
| and this sort of change would be a huge cause of concern. If you
| primarily vibe-code, this is not a problem but understand that
| they're making their product worse on purpose.
___________________________________________________________________
(page generated 2026-07-22 18:02 UTC)