@dorianmariefr if you wish for this feature to be considered, you'll need to expand a bit on your description. Like providing a few snippets of code that show how it is helpful, how often and how it makes your code better. Bonus point if you can link to open source code that would benefit from this.
Once that is said, I personally wanted this a bunch of time. As mentioned by others, in most case hash.delete(:foo) || default works fine, but sometimes explicit nil or false may be expected, in which case you have to use hash.delete(:foo) if hash.key?(:foo).
Also as @nobu (Nobuyoshi Nakada) mentioned, Hash#delete already handle a "default block" like fetch, so I think it would make sense to accept a positional default too, just for consistency's sake.