You can use <U+007B> for the left curly bracket.
@startuml
Alice -> Bob: [["this is the tooltip" <U+007B>this is the text starting with a curly bracket}]]
@enduml
Idea using a macro:
@startuml
!function linktext(txt)
!if (%substr(txt,0,1) == "{")
!return "<U+007B>" + %substr(txt,1,%strlen(txt)-1)
!else
!return txt
!endif
!endfunction
Alice -> Bob: [["this is the tooltip" linktext("{this is the text starting with a curly bracket}")]]
@enduml